MP4/H.264 codec was designed to provide high quality video, with minimal file sizes. It's consumer profile is typically an interframe format, using I (or keyframes), P, and B frames. Only the keyframe is unique, all other frames needing to be derived. See this
article for basic info.
Codecs such as consumer MP4/H.264 creates some unique issue for frame accurate editing. For instance, if you edit video frames which is not an I frame, the editor will need to both decompress and derive the frames needed between the keyframes. Apply the edit or effects to the frames, then re-compress the frames back. This is a very hardware intensive process. It's also why H.264 is not a recommended source for editing, and the edit process itself can introduce visible artifacts due to a continual process of compressing/decompressing the video while editing.
This is also why pros needing to edit interframe formats such as H.264, will typically convert the files to visual lossless intraframe formats such as Apple ProRes or Avid DNxHD, which essentially contain all I frames. Though technically lossy formats, uses less storage than uncompressed. And has the advantage of being less hardware intensive, though requires much faster drive access speeds.
The MP4/H.264 codec was ideal for use in consumer camera acquisition, and HD delivery due to lower bandwidth requirements. However, it does have other drawbacks such as a reduced 4:2:0 color space. Most pro camera operators prefer an intraframe format with higher color space such as 4:2:2. Especially important to maintain color info if grading (color correcting). Apple ProRes and Avid DNxHD being some popular examples. Though camera manufacturers also support their own codecs.
Note that H.264 supports different profiles which can offer lossless, intraframe, and higher color spaces. None of which you'll likely find supported natively in consumer cameras.