There are lots of cab ride videos on youtube, which is rather neat.
(Although there are some peculiar gaps in the coverage. Try as I might, I haven't been able to find one of the complete "Berks & Hants" route from Reading to Taunton. I've found one from Reading to Westbury, but nothing beyond Westbury, nor one with the Westbury and Frome avoiding lines. Anyone who has been more successful in their searching is invited to get in touch...)
But with far too many of them, there is a shitey problem. Every fifth frame is a duplicate. It looks as if some thick cunt has "converted" them from 25fps - or even 24fps, sometimes; I mean what the fuck? - to 30fps by inserting duplicate frames. I don't know if this is youtube being a cunt or people using shitty recording devices, though my guess would be it's youtube, because youtube is fucking American and far too much American stuff fails to acknowledge that the rest of the world doesn't necessarily follow their shitty standards; anyway, it is shit, because the duplicate frames make the video jerky and it is really fucking obvious.
It's also shit because what the fuck are they playing at using such a mindlessly crap method of frame rate conversion for in the first place now that we have all digital an' shit? They don't cough at the amount of processing power needed to transcode everything into VP9, which is incredibly slow at encoding, so surely for fuck's sake they can spare a few cycles to do proper interpolation instead of this cheap-ass crappy jerky duplicate frame bollocks.
The obvious way to sort this out is to use ffmpeg to knock out the duplicate frames and put the frame rate back to 24 or 25 as appropriate. The mpdecimate filter, which knocks out duplicate frames, is widely recommended for doing this. There are also one or two other filters which are supposed to be a bit more intelligent than mpdecimate in that they're intended for knocking out duplicate frames that occur at regular intervals because some stupid bastard has "converted" the frame rate by inserting duplicate frames, ie. the very problem under consideration. So really you'd think it ought to be quite easy.
But it isn't. It is in fact a complete dog's cunt.
The biggest problem is that ffmpeg fights like a total bastard against the change of frame rate. There appears to be no way to say to it "look, forget about what fucking frame rate it's "supposed" to be, just take this filtered sequence of frames and assume that each one comes 1/24 of a second after the one before no matter what". I have pissed about endlessly with the setpts and fps filters, and -r, and -vsync, and fuck knows what else, but no matter what I do I can't prevent ffmpeg from getting its own ideas and fucking things up. It insists on thinking something is out of sync somewhere and duplicating or dropping more frames to get it what it thinks is straight again. Basically, I set up filters to put things right and then ffmpeg takes the filter output and reintroduces exactly the fucking same kind of shitey corruption that I've just got rid of. Bastard.
The second biggest problem is that whenever the train stops at a station, there is a stretch where the only difference between one frame and the next is what you get from noise, and the duplicate frame detection filters get confused all to fuck. Not only is this bad in itself, it causes further confusion to ffmpeg's fighting over the frame rate, so all sorts of weird shit happens.
So I have devised my own method, which is a cumbersome pain in the arse, but at least it works. It goes like this:
First, sledgehammer the bastard thing into definitely and completely dropping every possible suggestion of a frame rate. This is done by separating the audio from the video so it can't use the video as a time reference, decomposing the video into individual frames, and storing them as .PNG files in a directory. Note that absolutely no filtering is done at this stage. It goes something like this:
ffmpeg -v verbose -y -i Cab_Ride_-_CST_to_CST_via_Erith_loop_-_020915-SZDTNVtfs6U.webm -an -vcodec png pngs/file-%07d.png ffmpeg -i Cab_Ride_-_CST_to_CST_via_Erith_loop_-_020915-SZDTNVtfs6U.webm -vn -acodec copy separated_audio.ogg
Next, I wrote a little program (see here) to scan through all those .PNG files, and for each one, to print the filename, a "coefficient of difference" between that file and the one before, and the word "DELETE" if that coefficient is below a certain threshold value. An invocation might be:
pngdircmp pngs > delete_list.txt
and the output of that program (provided on stdout, and here redirected to a text file) looks like this:
...... file-0062692.png 23756390 file-0062693.png 24161644 file-0062694.png 22924619 file-0062695.png 71280 DELETE file-0062696.png 18787681 file-0062697.png 18433107 file-0062698.png 18700722 file-0062699.png 18512712 file-0062700.png 153831 DELETE file-0062701.png 18685303 file-0062702.png 18903868 file-0062703.png 19401990 file-0062704.png 19828520 file-0062705.png 274744 DELETE file-0062706.png 20459098 file-0062707.png 20843192 file-0062708.png 20180684 ......
The numbers in the second column are the "coefficients of difference". In this snippet the much lower coefficients of the duplicate frames are obvious, but when the train isn't moving things are much less clear:
...... file-0031240.png 257927 DELETE file-0031241.png 67880 DELETE file-0031242.png 321328 DELETE file-0031243.png 413395 DELETE file-0031244.png 275943 DELETE file-0031245.png 383397 DELETE file-0031246.png 112941 DELETE file-0031247.png 357568 DELETE file-0031248.png 475578 DELETE file-0031249.png 437902 DELETE file-0031250.png 164117 DELETE file-0031251.png 114789 DELETE file-0031252.png 197019 DELETE file-0031253.png 233254 DELETE file-0031254.png 326573 DELETE file-0031255.png 309447 DELETE file-0031256.png 0 DELETE file-0031257.png 225321 DELETE file-0031258.png 308629 DELETE file-0031259.png 333241 DELETE file-0031260.png 264233 DELETE ......
That file, therefore, is not the whole story; if you look through it it is apparent that there are several places, like the above, where it's fucked up. In stations there are lots of files marked DELETE one after the other. There are also glitches every thousand files or so because of this stupid crap about the 30 in "30fps" actually being 29.97 (for fuck's sake; and no, there isn't a "good reason" for it, there's a shit reason, which is that NTSC is a shitty bodge and the 29.97 is part of the shitty bodginess. PAL, which doesn't have the problem, is how you do it properly). So the next stage is to post-process it with another little program (see here), which tries to follow the cyclic pattern of "correct" deletions and makes a pretty good guess at what to do when it finds an anomaly. Which would go like:
pngdelchk delete_list.txt > delete_list_2.txt
and produce output looking like:
...... file-0062692.png 23756390 2 1 file-0062693.png 24161644 5 2 file-0062694.png 22924619 3 3 file-0062695.png 71280 0 4 DELETE file-0062696.png 18787681 3 0 file-0062697.png 18433107 1 1 file-0062698.png 18700722 4 2 file-0062699.png 18512712 3 3 file-0062700.png 153831 0 4 DELETE file-0062701.png 18685303 3 0 file-0062702.png 18903868 4 1 file-0062703.png 19401990 5 2 file-0062704.png 19828520 5 3 file-0062705.png 274744 0 4 DELETE file-0062706.png 20459098 5 0 file-0062707.png 20843192 6 1 file-0062708.png 20180684 4 2 ......
Here there are two extra columns to give a bit more clue as to what's going on. The first extra column is how many frames nearby on either side have a lower coefficient than this frame. The second is where the program thinks we've got to in the cycle.
As one would hope, it hasn't changed that bit of the file since it was OK to start with; compare what it does on the other sample:
...... file-0031240.png 257927 4 3 file-0031241.png 67880 0 4 DELETE file-0031242.png 321328 4 0 file-0031243.png 413395 6 1 file-0031244.png 275943 3 2 file-0031245.png 383397 5 3 file-0031246.png 112941 0 4 DELETE file-0031247.png 357568 2 0 file-0031248.png 475578 6 1 file-0031249.png 437902 5 2 file-0031250.png 164117 2 3 file-0031251.png 114789 0 4 DELETE file-0031252.png 197019 2 0 file-0031253.png 233254 3 1 file-0031254.png 326573 6 2 file-0031255.png 309447 5 3 file-0031256.png 0 0 4 DELETE file-0031257.png 225321 1 0 file-0031258.png 308629 3 1 file-0031259.png 333241 6 2 file-0031260.png 264233 3 3 ......
Bleedin' magic, innit.
It isn't completely perfect, but it seems to only be out by a couple of hundred frames in 100,000, so I think we can live with that. It's certainly a fucking sight better than what ffmpeg's filters manage even before its refusal to let me manage the frame rate introduces new fuckups.
In the context of the example video, it has changed it from juddering all the bloody time so you can't not see it and it's fucking annoying, to only giving the occasional judder every now and then which is much less annoying than the other things wrong with the video, like the step changes in exposure as the light level varies or when it starts raining at Falconwood. I've more than half an idea that most if not all of the remaining visible judders are the actual camera being shit anyway.
It would of course be perfectly possible to combine these two programs into one, but having them separate makes it easier to see what's going on. It's also useful to keep the actual comparison part of the process, which is very slow, as simple and unintelligent as possible, so that bit can be done once with confidence that it won't have to be done again, and keep the experimentation with sorting out the anomalies to a separate stage, which is very fast so it doesn't matter about having to fuck about doing it again (for example, if at some future time I come across a video which the post-processor doesn't make a very good fist of, and so have to rejigger the code).
Having corrected the file, it's then straightforward to delete the unwanted files. It is also necessary to rename the files back into a continuous numerical sequence after the deletion has left gaps in it, otherwise ffmpeg will ignore all the files after the first gap (did I mention it was a cunt?) There are of course craploads of ways to do this with bash one-liners and fuck knows what else, but I wrote another little program (see here) to do it, mainly because bash scripts that loop through 100,000 files are annoyingly slow under any circumstances. So it goes like this:
pngcleanup pngs delete_list_2.txt
The cleaned-up set of .PNG files can then be converted back into a video, with the frame rate set to 24fps and nothing to make ffmpeg think it isn't getting that. Like, say, for instance:
ffmpeg -v verbose -y -f image2 -vcodec png -r 24 -i pngs/file2-%07d.png -pix_fmt yuv420p -vcodec libx264 -vb 3000000 -r 24 reconstituted_video.mp4
And finally, the video and the audio can be stuck back together in the same file. There is still a slight wriggle required because the few remaining uncorrected frame deletion errors mean the video and audio files now differ in length by a few seconds. This, it seems, can be dealt with quite easily by multiplying or dividing the audio sample rate by the ratio of the two lengths to slow down or speed up the audio as required, then resampling it back to a conventional rate (which we would be doing anyway to synchronise it with the video's new timestamps). In the example I'm using here, the duration of the corrected video came out at 01:13:46.71, while the audio duration was 01:13:41.96, so the compensation was to change the audio sample rate from 44100 to 44053.
ffmpeg -v verbose -y -i reconstituted_video.mp4 -i separated_audio.ogg -vcodec copy -af asetrate=r=44053,aresample=resampler=swr:async=5000:min_comp=0.02 -acodec aac Cab_Ride_-_CST_to_CST_via_Erith_loop_-_020915-SZDTNVtfs6U.fixed.mp4
Here follow the various "little programs" referred to earlier. Note that these are quick hacks, not robust code; they are light on the error checking and make various unstated assumptions such as that filenames are less than 256 characters long, etc. So don't blame me if they explode or make your computer turn into a Thing from the Deep.
Here is the program for comparing all the .PNG files. Optional parameters are an integer for the threshold value, and a string for the directory to look at. If they are omitted, it uses the current directory and a threshold of 1000000. It uses the libpnglite library to decode the .PNG files because it's a sight less fucking about than the usual fully-complex libpng needs.
Download: pngdircmp.c.gz
/* * Uses: libpnglite * Link with: -lpnglite -lm */ #define _GNU_SOURCE #define _POSIX_C_SOURCE 200809L #include <stdio.h> #include <stdlib.h> #include <string.h> #include <ctype.h> #include <dirent.h> #include <errno.h> #include <math.h> #include <pnglite.h> int rms(unsigned char *dat1, unsigned char *dat2, unsigned len) { int n, d; double t; t = 0.0; for (n = 0; n < len; n++) { d = (int)(*dat1++) - (int)(*dat2++); t += (double)(d * d); } t /= (double)len; return((int)(sqrt(t) * 1000000.0)); } int pngsuff(const struct dirent *d) { const char *s; s = d->d_name; s += (strlen(s) - 4); return(strncasecmp(s, ".png", 4) ? 0 : 1); } int main(int argc, char *argv[]) { png_t png1, png2; unsigned dsz1, dsz2; unsigned char *dat1, *dat2, *dat3; int e, t, n, x; int ndirs; struct dirent **ents; struct dirent *d; char *dir, *s, *fn; dir = NULL; t = 0; for (x = 1; x < argc; x++) { n = 1; for (s = argv[x]; *s; s++) if (!isdigit(*s)) n = 0; if (n) { if (sscanf(argv[x], "%d", &t) != 1) { fprintf(stderr, "Error parsing number %s\n", argv[x]); exit(1); } } else { dir = argv[x]; } if ((dir) && (t)) break; } if (!dir) dir = "."; fn = malloc(strlen(dir) + 258); if (!t) t = 1000000; if ((ndirs = scandir(dir, &ents, pngsuff, versionsort)) < 0) { fprintf(stderr, "Error reading directory %s: %s\n", dir, strerror(errno)); exit(1); } if (!ndirs) { fprintf(stderr, "Directory %s contains no .png files\n", dir); exit(1); } png_init(0, 0); memset(&png1, 0, sizeof(png_t)); d = ents[0]; sprintf(fn, "%s/%s", dir, d->d_name); if ((e = png_open_file_read(&png1, fn)) != PNG_NO_ERROR) { fprintf(stderr, "Error reading %s: %s\n", fn, png_error_string(e)); exit(1); } dsz1 = png1.width * png1.height * png1.bpp; dat1 = malloc(dsz1); dat2 = malloc(dsz1); if ((e = png_get_data(&png1, dat1)) != PNG_NO_ERROR) { fprintf(stderr, "Error reading data from %s: %s\n", fn, png_error_string(e)); exit(1); } png_close_file(&png1); printf("%s\t%d\n", d->d_name, t + 1); for (n = 1; n < ndirs; n++) { d = ents[n]; memset(&png2, 0, sizeof(png_t)); sprintf(fn, "%s/%s", dir, d->d_name); if ((e = png_open_file_read(&png2, fn)) != PNG_NO_ERROR) { fprintf(stderr, "Error reading %s: %s\n", fn, png_error_string(e)); continue; } dsz2 = png2.width * png2.height * png2.bpp; if (dsz1 != dsz2) continue; if ((e = png_get_data(&png2, dat2)) != PNG_NO_ERROR) { fprintf(stderr, "Error reading data from %s: %s\n", fn, png_error_string(e)); continue; } png_close_file(&png2); e = rms(dat1, dat2, dsz1); printf("%s\t%d%s\n", d->d_name, e, e < t ? "\tDELETE" : ""); dat3 = dat1; dat1 = dat2; dat2 = dat3; } }
Here is the program for post-processing the list of deletions. Its one, required, parameter is the filename of the list.
Download: pngdelchk.c.gz
/* * Link with: -lm */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <math.h> #define FRINGE 3 #define FRN ((FRINGE << 1) + 1) struct lion { char *fn; int n; int nl; int del; int mark; }; int readfile(char *fn, void **data, struct lion **lions, struct lion **maxlion) { int len; int n, x, a, b, c, i; char *s, *e; FILE *fp; struct lion *l; if ((fp = fopen(fn, "r")) == NULL) { fprintf(stderr, "%s: cannot open\n", fn); exit(1); } fseek(fp, 0, SEEK_END); len = ftell(fp); fseek(fp, 0, SEEK_SET); *data = malloc(len + 2); s = (char *)*data; n = 0; while (fgets(s, 256, fp)) { s += strlen(s); if (*(s - 1) == '\n') n++; } fclose(fp); e = s; strcat(s, "x\n"); s = (char *)*data; s = strtok(s, ", \t\n"); *lions = malloc((n + 1) * sizeof(struct lion)); l = *lions; x = 0; while ((s) && (s < e)) { l->fn = s; s = strtok(NULL, ", \t\n"); if (sscanf(s, "%d", &(l->n)) != 1) l->n = 0; s = strtok(NULL, ", \t\n"); if (l->del = !strncmp(s, "DELETE", 6)) s = strtok(NULL, ", \t\n"); l->mark = 0; l++; } *maxlion = l; a = 0; c = FRINGE; for (x = 0; x < n; x++) { i = 0; for (b = a; b < c; b++) if ((*lions)[b].n < (*lions)[x].n) i++; (*lions)[x].nl = i; if (x > FRINGE) a++; if (c < n) c++; } return(n); } int avgspc(struct lion *l, int n) { int x, t, a, i; t = 0; a = 0; i = 0; for (x = 0; x < n; x++, l++) { if ((l->del) && (a > (FRINGE - 1))) { t += a; a = 0; i++; } else { a++; } } return((int)lround((double)t / (double)i)); } int main(int argc, char *argv[]) { void *data; struct lion *lions, *l, *lmax, *ll; struct lion *f[FRN]; int n, x; int t, a, i; if (argc < 2) { fprintf(stderr, "No filename given\n"); exit(1); } n = readfile(argv[1], &data, &lions, &lmax); a = avgspc(lions, n); for (l = lions; !(l->del); l++); i = 0; for (; l < lmax; l++, i++) { l->mark = i; if ((i < a) && (!(l->del))) { if (!(l->n)) { l->del = 1; i = -1; } continue; } if ((i == a) && (l->del)) { ll = l; ll++; if ((!(ll->n)) || (!(ll->nl))) { l->del = 0; i--; continue; } i = -1; continue; } if ((i < a) && (l->del)) { if (!(l->n)) { i = -1; continue; } if (i < FRINGE) { l->del = 0; continue; } if (!(l->nl)) { i = -1; continue; } l->del = 0; continue; } if ((i >= a) && (!(l->del))) { if ((!(l->n)) || (!(l->nl))) { l->del = 1; i = -1; } continue; } if ((i > a) && (l->del)) { if ((!(l->n)) || (!(l->nl))) { i = -1; continue; } l->del = 0; i = 0; continue; } } for (l = lions; l < lmax; l++) { printf("%s\t%8d%3d%3d%s\n", l->fn, l->n, l->nl, l->mark, l->del ? "\tDELETE" : ""); } }
Here is the program for deleting the unwanted .PNG files and renaming those that remain back into a continuous numerical sequence. Its parameters are the directory of .PNG files and the filename of the list of deletions, in either order.
Download: pngcleanup.c.gz
/* * Link with: -lm */ #define _GNU_SOURCE #define _POSIX_C_SOURCE 200809L #include <stdio.h> #include <stdlib.h> #include <string.h> #include <ctype.h> #include <dirent.h> #include <errno.h> #include <fcntl.h> #include <sys/types.h> #include <sys/stat.h> #include <unistd.h> #include <math.h> #define NCH 300 int pngsuff(const struct dirent *d) { const char *s; s = d->d_name; s += (strlen(s) - 4); return(strncasecmp(s, ".png", 4) ? 0 : 1); } int main(int argc, char *argv[]) { struct stat sb; struct dirent **ents; int d, e, n, x; char *dir = NULL; char *fn = NULL; FILE *fp; char buf[NCH]; char *qfn1, *qfn2; char *s; if (argc != 3) { fprintf(stderr, "Usage: pngcleanup dir file\n" " pngcleanup file dir\n"); exit(1); } for (x = 1; x <= 2; x++) { if (stat(argv[x], &sb) < 0) { fprintf(stderr, "%s: cannot stat(): %s\n", argv[x], strerror(errno)); exit(1); } if (S_ISREG(sb.st_mode)) { fn = argv[x]; } else if (S_ISDIR(sb.st_mode)) { dir = argv[x]; } else { fprintf(stderr, "%s is neither a file nor a directory\n", argv[x]); exit(1); } } if ((!fn) || (!dir)) { fprintf(stderr, "Usage: pngcleanup dir file\n" " pngcleanup file dir\n"); exit(1); } if ((fp = fopen(fn, "r")) == NULL) { fprintf(stderr, "%s: cannot open: %s\n", fn, strerror(errno)); exit(1); } qfn1 = malloc(strlen(dir) + NCH); qfn2 = malloc(strlen(dir) + NCH); d = e = n = 0; while (fgets(buf, 256, fp)) { n++; for (s = buf; *s && (*s != '\n'); s++); *s = 0; for (s = buf; *s && !isblank(*s); s++); *s++ = 0; s += (strlen(s) - 6); if (!strncmp(s, "DELETE", 6)) { sprintf(qfn1, "%s/%s", dir, buf); if (unlink(qfn1) < 0) { fprintf(stderr, "%s: cannot unlink(): %s\n", qfn1, strerror(errno)); e = 1; } else { d++; } } } fclose(fp); if ((e) || (n - d <= 0)) { fprintf(stderr, "Error(s) deleting files - not renaming\n"); exit(1); } for (s = buf; *s && !isdigit(*s); s++); if (!*s) { fprintf(stderr, "Cannot find numeric portion in %s\n", buf); exit(1); } sprintf(s, "%%0%dd.png", (int)(ceil(log10((double)n))) + 1); if ((n = scandir(dir, &ents, pngsuff, versionsort)) < 0) { fprintf(stderr, "Error reading directory %s: %s\n", dir, strerror(errno)); exit(1); } if (!n) { fprintf(stderr, "Directory %s contains no .png files\n", dir); exit(1); } sprintf(qfn2, "%s/", dir); s = qfn2 + strlen(qfn2); for (x = 0; x < n; x++) { sprintf(qfn1, "%s/%s", dir, (ents[x])->d_name); sprintf(s, buf, x + 1); if (rename(qfn1, qfn2) < 0) { fprintf(stderr, "Cannot rename %s to %s: %s: aborting\n", qfn1, qfn2, strerror(errno)); exit(1); } } }
Back to Pigeon's Nest
Be kind to pigeons