pyTivo Discussion Forum Forum Index pyTivo Discussion Forum
Answers and the development of pyTivo a TiVo transcoding server
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Remux without reencoding for MKV, AVCHD, etc. in Push
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
 
Post new topic   Reply to topic    pyTivo Discussion Forum Forum Index -> pyTivo
 View previous topic :: View next topic  
Author Message
Iluvatar



Joined: 29 Feb 2008
Posts: 335

PostPosted: Mon May 30, 2011 7:53 pm    Post subject: Reply with quote

so here are the changes I have made to allow pyTivo to remux any video with incompatible audio into mp4 for streaming as long as it has a compatible h264 video stream. This should detect the incompatible audio and re-encode as necessary...ex: DTS, vorbis, etc... You may still run into the non-monotone timestamp error which will still cause a full transcode. FFmpeg 0.6.3 doesn't seem to have this issue but later and earlier revisions do.

Most of this code already existed in pyTivo but I wasn't sure how to create a diff so I'll just post the affected sections.

transcode.py - I have split the mp4_remuxable into separate audio video tests and edited mp4_remux function

Code:

# this was previously just a single function that would fail on audio check if incompatible
def mp4_remuxable_video(inFile, tsn=''):
    vInfo = video_info(inFile)
    return (tivo_compatible_video(vInfo, tsn, 'video/mp4')[0])

def mp4_remuxable_audio(inFile, tsn=''):
    vInfo = video_info(inFile)
    return (tivo_compatible_audio(vInfo, inFile, tsn, 'video/mp4')[0])

# most of this is wmcbrine's code.
def mp4_remux(inFile, basename, tsn=''):
    outFile = inFile + '.pyTivo-temp'
    newname = basename + '.pyTivo-temp'
    if os.path.exists(outFile):
        return None  # ugh!

    ffmpeg_path = config.get_bin('ffmpeg')
    fname = unicode(inFile, 'utf-8')
    oname = unicode(outFile, 'utf-8')
    if mswindows:
        fname = fname.encode('iso8859-1')
        oname = oname.encode('iso8859-1')

# I have only added this audio check call with ffmpeg cmd to re-encode

    if mp4_remuxable_audio(inFile, tsn):
       debug('remuxing ' + inFile + ' to ' + outFile)
       cmd = [ffmpeg_path, '-i', fname, '-vcodec', 'copy', '-acodec',
             'copy', '-f', 'mp4', oname]
    else:
       debug('remuxing with re-encoded audio ' + inFile + ' to ' + outFile)
       cmd = [ffmpeg_path, '-i', fname, '-vcodec', 'copy', '-ab', '448k',
            '-ar', '48000', '-acodec', 'ac3', '-copyts', '-f', 'mp4', oname]
       
    ffmpeg = subprocess.Popen(cmd)
   
    if ffmpeg.wait():
        debug('error during remuxing')
        os.remove(outFile)
        return None

    return newname


video.py - I have adjusted the mp4_remux(able) functions accordingly to work with the new code.

Code:

if (mime == 'video/mpeg' and
                transcode.mp4_remuxable_video(f['path'], f['tsn'])):
                new_path = transcode.mp4_remux(f['path'], f['name'], f['tsn'])
                if new_path:
                    mime = 'video/mp4'
                    f['name'] = new_path


I have attached the full .py files and they should be able to be dropped in on wmcbrines git from 6/25/11.

I have tested this with many of the different types of files I have and they have all worked. However this is not to say you would have different results. I am sure someone will have issues with the hard coded ffmpeg command but I really didn't have time to hash that one out. Maybe another time.

edit: i have updated my the post to add comments in the code to show exactly what I changed. Most of what you see there already existed.

edit: updated files to match newer revision on wmcbrine's branch.

edit: deleted attachment. lucasnz's post below seems to have much better solution than what I had worked out.


Last edited by Iluvatar on Thu Jul 07, 2011 11:37 am; edited 9 times in total
Back to top
View user's profile Send private message
jcthorne



Joined: 30 Jan 2008
Posts: 243
Location: Houston, TX

PostPosted: Tue May 31, 2011 9:21 am    Post subject: Reply with quote

Your additions seem to assume that any h264 video stream is playable by the tivo as is. This is not the case. The AVC profile must be 4.1 or less, for 1080 resolution, must be 4 ref frames or less. Also, for a series 3 unit, 720p video must be full frame, ie 1280x720.

Also, audio bitrate need not be limited to 448kbps any longer. 640 seems to work fine now on both TivoHD and Premiere.

Further, the resultant files that ffmpeg creates from this command line are not IMSA standard streamable and the tivo will not accept them as is. The MOV atom must be at the front of the file.
Back to top
View user's profile Send private message
Iluvatar



Joined: 29 Feb 2008
Posts: 335

PostPosted: Tue May 31, 2011 1:23 pm    Post subject: Reply with quote

I understand that Tivo's will not accept higher than 4.1 AVC profile however I have done no additional checks other than what wmcbrine has programmed pyTivo to check for. If the video passed before then it will pass now (whether TiVo accepts the file or not). I am aware of the 1:1 PAR issue with S3/HD units as I frequently run up against it but pyTivo never fixed this anyways (on pushes). All I have done is moved the audio checks into the remux function in order to prevent the entire file from being transcoded which is a waste of time. Most of what you see there is from wmcbrine. He programmed the original MP4 remux function...I just made it slightly more compatible....to fit my needs.

As to the mov atom, pyTivo already checks for this once it exits the remux function and calls qtfaststart.py if necessary. I have had zero issues (except the h264 level 5 video, but pyTivo doesnt check this anyways as ffmpeg does not return this info) with streaming MKV files with DTS or other audio. I have thrown at least 30 files at my premiere and they work perfectly.

As to the audio, I have hardcoded the ffmpeg command but only because I was lazy. I'm certain it can be changed to use the user defined ffmpeg overrides in the pytivo.conf. Feel free to change it to whatever you want. To be honest it works for me and I just assumed that since I am not a programmer these are probably just ugly hacks that wouldn't ever make it into wmcbrine's GIT unless somebody took it and brought it up to standards.
Back to top
View user's profile Send private message
wmcbrine



Joined: 04 Jan 2008
Posts: 2009
Location: Maryland

PostPosted: Tue May 31, 2011 4:14 pm    Post subject: Reply with quote

jcthorne wrote:
Further, the resultant files that ffmpeg creates from this command line are not IMSA standard streamable and the tivo will not accept them as is. The MOV atom must be at the front of the file.

Yes, that's why this is written out to a temp file: so we can run qtfaststart on it during the transfer. Unfortunately I know of no way to create a streamable MP4 in one pass.

_________________
My pyTivo fork . My page
Back to top
View user's profile Send private message Visit poster's website
ercdvs



Joined: 11 Nov 2008
Posts: 8

PostPosted: Tue Jun 07, 2011 3:31 pm    Post subject: Reply with quote

I had not seen this mentioned, but does this feature respect the settings on a global or per tivo basis ?

ie will it always attempt a straight copy if I have a _tivo_ setting of audio_ch = 2 set?

Im getting mixed results during my tests. it seems no matter what my settings, pytivo will do the .pytivo-temp version, and if i kill ffmpeg, it will restart itself using my config settings while trying to stream directly.

basically i am finding that a copy takes just a few minutes, then transfers @ 20+ Mb/sec.. while my 'lower quality' transcode stream takes a lot longer, and seems to transfer @ 6 Mb/sec max (but strangely seems to give both -ac 2 AND -acodec copy:

/usr/local/bin/ffmpeg -i /home/master/video/TV/vidfile.mkv -vcodec mpeg2video -b 8388k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -ac 2 -acodec copy -copyts -threads 2 -f vob -
Back to top
View user's profile Send private message
Iluvatar



Joined: 29 Feb 2008
Posts: 335

PostPosted: Tue Jun 07, 2011 5:29 pm    Post subject: Reply with quote

ercdvs wrote:
I had not seen this mentioned, but does this feature respect the settings on a global or per tivo basis ?

ie will it always attempt a straight copy if I have a _tivo_ setting of audio_ch = 2 set?

Im getting mixed results during my tests. it seems no matter what my settings, pytivo will do the .pytivo-temp version, and if i kill ffmpeg, it will restart itself using my config settings while trying to stream directly.

basically i am finding that a copy takes just a few minutes, then transfers @ 20+ Mb/sec.. while my 'lower quality' transcode stream takes a lot longer, and seems to transfer @ 6 Mb/sec max (but strangely seems to give both -ac 2 AND -acodec copy:

/usr/local/bin/ffmpeg -i /home/master/video/TV/vidfile.mkv -vcodec mpeg2video -b 8388k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -ac 2 -acodec copy -copyts -threads 2 -f vob -


No the code does not change the audio stream to user defined settings it only copies what is in the container as long as it is compatible. The edited version I posted above transcodes incompatible audio but still does not change the stream to the user defined settings but only changes the codec to AC3 at a bitrate of 448kbps.

Why are you killing ffmpeg? It is remuxing the streams into a compatible container. When you kill ffmpeg pyTivo assumes it errored out and then does a normal full transcode on the file (along with your user defined audio settings)

The way the code is now you can either have a fast copy remux with audio the same as it is in the original file or you can do a full slow transcode to get your custom audio settings.

I have a feeling it wouldn't be hard to do but when I posted my edits I did not have time to figure out how to use pyTivo's existing functions to do this. Maybe wmcbrine will take a stab at it if he is interested.
Back to top
View user's profile Send private message
ercdvs



Joined: 11 Nov 2008
Posts: 8

PostPosted: Tue Jun 07, 2011 7:50 pm    Post subject: Reply with quote

I was killing ffmpeg because i was testing various options out, starting a pull, then killing it after i saw what it was sending. it was just curious to see a 'backup' transcode when the main process failed.

Yeah, 99% of the time sending the full file over in a different container is much better, even though with this particular tv HD video and 5.1 audio is wasted.

All my settings and test are still valid for transcoded streams though
Back to top
View user's profile Send private message
lucasnz



Joined: 13 Sep 2010
Posts: 251

PostPosted: Thu Jul 07, 2011 9:22 am    Post subject: Reply with quote

I've tweaked wmcbrine's code to remux using some of the standard algorithms already in transcode.py. It will also transcode the audio track if required.

See the diff here;
http://repo.or.cz/w/pyTivo/wmcbrine/lucasnz.git/commitdiff/f65a7e7d81d51eaf456e0387e7d7279c77650213
Back to top
View user's profile Send private message
Iluvatar



Joined: 29 Feb 2008
Posts: 335

PostPosted: Thu Jul 07, 2011 11:35 am    Post subject: Reply with quote

lucasnz wrote:
I've tweaked wmcbrine's code to remux using some of the standard algorithms already in transcode.py. It will also transcode the audio track if required.

See the diff here;
http://repo.or.cz/w/pyTivo/wmcbrine/lucasnz.git/commitdiff/f65a7e7d81d51eaf456e0387e7d7279c77650213


This is great. Thank you for your efforts. Exactly what I wanted but didn't know how to code. I'll throw lots of videos at it and let you know how it worked out.
Back to top
View user's profile Send private message
wmcbrine



Joined: 04 Jan 2008
Posts: 2009
Location: Maryland

PostPosted: Thu Jul 07, 2011 11:54 am    Post subject: Reply with quote

Note that, by using select_audiocodec(), your patch will transcode otherwise-compatible AAC tracks.
_________________
My pyTivo fork . My page
Back to top
View user's profile Send private message Visit poster's website
Iluvatar



Joined: 29 Feb 2008
Posts: 335

PostPosted: Thu Jul 07, 2011 12:42 pm    Post subject: Reply with quote

So far works great. DTS audio and >448k AC3 is transcoded.

I would guess though that you could remove the following code which seems redundant given the function:

Code:

'video_br': select_videobr(inFile, tsn),
'video_fps': select_videofps(inFile, tsn),
'max_video_br': select_maxvideobr(tsn),
'buff_size': select_buffsize(tsn),
'aspect_ratio': ' '.join(select_aspect(inFile, tsn)),


To get to this point the file must have already passed video checks so the video at this point is always going to be '-vcodec copy' and the above settings are never ever used. If the remux errors out then the file still gets returned back to the normal pyTivo transcode function anyways.

edit: nevermind, it looks like if 'getFFmpegTemplate(tsn)' doesn't get all variables it errors out.
Back to top
View user's profile Send private message
txporter



Joined: 11 Dec 2008
Posts: 263
Location: Austin, TX

PostPosted: Thu Jul 07, 2011 7:06 pm    Post subject: Reply with quote

wmcbrine wrote:
Note that, by using select_audiocodec(), your patch will transcode otherwise-compatible AAC tracks.


You are saying that H.264 video with AAC audio will transcode with this latest patch? Not good. I transcode everything to AAC so that I don't have to re-encode audio for my idevices.
Back to top
View user's profile Send private message
lucasnz



Joined: 13 Sep 2010
Posts: 251

PostPosted: Thu Jul 07, 2011 8:04 pm    Post subject: Reply with quote

txporter wrote:
wmcbrine wrote:
Note that, by using select_audiocodec(), your patch will transcode otherwise-compatible AAC tracks.


You are saying that H.264 video with AAC audio will transcode with this latest patch? Not good. I transcode everything to AAC so that I don't have to re-encode audio for my idevices.


Ths is by design. I find the tivo gets the channels mixed up with 6 channel aac audio.... So up to now my process has been to manually transcode aac audio to ac3.
Back to top
View user's profile Send private message
txporter



Joined: 11 Dec 2008
Posts: 263
Location: Austin, TX

PostPosted: Thu Jul 07, 2011 8:25 pm    Post subject: Reply with quote

Is there a way to not get that by flag or something? I run my pytivo instances off of a tonidoplug, and I don't think it is really up to transcoding video on the fly.
Back to top
View user's profile Send private message
lucasnz



Joined: 13 Sep 2010
Posts: 251

PostPosted: Thu Jul 07, 2011 8:44 pm    Post subject: Reply with quote

txporter wrote:
Is there a way to not get that by flag or something? I run my pytivo instances off of a tonidoplug, and I don't think it is really up to transcoding video on the fly.


No just the audio will be transcoded the video is copied as per the -vcodec copy command.

If you have debuging enabled the cmd that it run is output to the log file....
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    pyTivo Discussion Forum Forum Index -> pyTivo All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
Page 2 of 7

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
Site is in NO WAY affiliated with TiVo Inc

Powered by phpBB © 2001, 2005 phpBB Group
phpBB SEO

Get pytivo at SourceForge.net. Fast, secure and Free Open Source software downloads
[ Time: 0.1554s ][ Queries: 12 (0.0137s) ][ GZIP on - Debug on ]