pyTivo Discussion Forum Forum Index pyTivo Discussion Forum
Answers and the development of pyTivo a TiVo transcoding server
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Adding multi-threaded decoding capability

 
Post new topic   Reply to topic    pyTivo Discussion Forum Forum Index -> pyTivo
 View previous topic :: View next topic  
Author Message
Iluvatar



Joined: 29 Feb 2008
Posts: 335
PostPosted: Thu Jan 12, 2012 2:46 am    Post subject: Adding multi-threaded decoding capability Reply with quote
Attached is patch that makes use of FFmpeg's multi-threaded decoding capability. pyTivo already uses multi-threaded encoding with the '-threads X' argument at the back end of FFmpeg's command but lacking was the same at the front end (ie. 'ffmpeg -threads X -i INPUT.....')

No guarantees that this works with old FFmpeg versions. I will test this soon.

I have seen overall increases of 15% when pulling to the TiVo. I only have a dual core CPU however. Increases may be better with a beefier CPU.

I created a new config option 'ffmpeg_threads' that is the same option as what used to be set in the ffmpeg_prams variable. I left the 'ffmpeg_prams' variable there in case there are other arguments added by the user. There doesn't seem to be any consequence of users unknowingly having both config options set to '-threads X', however I do recommend clearing your 'ffmpeg_prams' variable just in case.

I have been playing around with GIT and setup a fork of wmcbrines development which also integrates lucasnz's changes.

The patch is a little messy as I'm not very familiar with this kind of thing but feel free to take this patch if it interests you and do with what you want. Or download my pyTivo fork at: http://repo.or.cz/w/pyTivo/wmcbrine/taylor.git

Code:

From bb0c20bf7eaf5a427c1105a753791ecc0fd9eaa2 Mon Sep 17 00:00:00 2001
From: Taylor Spencer <taylorspencer@hotmail.com>
Date: Wed, 11 Jan 2012 16:43:25 -0500
Subject: [PATCH] add getFFmpeg_threads() function.  This allows the user to
 specifically set CPU threads for FFmpeg to use.  This
 needed to be split off from the old getFFmpegPrams()
 because thread usage can be used at the beginning of the
 FFmpeg command to increase threaded decoding performance.
 Default is no setting or (1) thread.

This is the first part.  Next will be integration into the FFmpeg command (i.e.. ffmpeg -threads x -i)
---
 config.py                 |   15 +++++++++++++++
 plugins/settings/help.txt |   16 ++++++++++++----
 2 files changed, 27 insertions(+), 4 deletions(-)

diff --git a/config.py b/config.py
index f719d2d..31efd22 100644
--- a/config.py
+++ b/config.py
@@ -281,6 +281,21 @@ def getFFmpegTemplate(tsn):
             %(audio_fr)s %(audio_ch)s %(audio_codec)s %(audio_lang)s \
             %(ffmpeg_pram)s %(format)s'
 
+def getFFmpegThreads():
+    if config.has_option('Server', 'ffmpeg_threads'):
+       #older FFmpeg builds have history of crashing if threads < 1
+       try:
+          threads = max(int(float(config.get('Server', 'ffmpeg_threads'))), 1)
+       except ValueError:
+          return 1
+
+       #threads max is 16
+       if threads <= 16:
+          return threads
+
+    #If option invalid or not specified, ffmpeg_threads should default to 1
+    return 1
+
 def getFFmpegPrams(tsn):
     return get_tsn('ffmpeg_pram', tsn, True)
 
diff --git a/plugins/settings/help.txt b/plugins/settings/help.txt
index 91f764d..7b8a80f 100644
--- a/plugins/settings/help.txt
+++ b/plugins/settings/help.txt
@@ -501,6 +501,16 @@ conflict could also occur if the source file has really corrupt
 sections.
 Example Settings: True, False
 Available In: Tivos, HD_tivos, SD_tivos
+
+ffmpeg_threads
+
+Default Setting: None
+Valid Entries: Any number up to 16 that represents the number of CPU threads FFmpeg has access to.
+Required: No
+Skill: Very Advanced
+Description:  Using multiple threads with FFmpeg if your CPU has them can speed up transcoding time.  Using a setting of '2' for a dual core CPU may show increased transcoding speeds of 10% or more.  If you have a dual core CPU with hyperthreading then the setting might be '4'.   There is a diminishing return for this setting so make sure to set this to no more than the amount of threads your CPU is capable of handling.  After a certain point the TiVo is the limiting speed factor.
+Example Settings: Any number from 1 to 16
+Available In: Server
 
 ffmpeg_pram
 
@@ -509,10 +519,8 @@ Valid Entries: A valid ffmpeg command
 Required: No
 Skill: Very Advanced
 Description: This allows you to append additional raw ffmpeg commands to
-the ffmpeg template. For example, you would enter '-threads 2' here if
-you have multiple processors and want ffmpeg to use both processors to
-speed up transcoding.
-Example Settings: -threads 2
+the ffmpeg template.
+Example Settings: A valid FFmpeg command
 Available In: Server, Tivos, HD_tivos, SD_tivos
 
 ffmpeg_tmpl
--
1.7.5.4

From 6de9db8ef543fcb1775fca8642b9a0968e4cecee Mon Sep 17 00:00:00 2001
From: Taylor Spencer <taylorspencer@hotmail.com>
Date: Wed, 11 Jan 2012 17:32:32 -0500
Subject: [PATCH] add select_ffmpegthreads() function in transcode.py and
 update the standard ffmpeg settings template to include
 threads setting for a pull.

TODO: At some point a check needs to be made for users who still have '-threads X' for ffmpeg_prams set in their config.
---
 config.py                  |    2 +-
 plugins/video/transcode.py |    4 ++++
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/config.py b/config.py
index 31efd22..315defc 100644
--- a/config.py
+++ b/config.py
@@ -279,7 +279,7 @@ def getFFmpegTemplate(tsn):
     return '%(video_codec)s %(video_fps)s %(video_br)s %(max_video_br)s \
             %(buff_size)s %(aspect_ratio)s %(audio_br)s \
             %(audio_fr)s %(audio_ch)s %(audio_codec)s %(audio_lang)s \
-            %(ffmpeg_pram)s %(format)s'
+            %(ffmpeg_pram)s %(ffmpeg_threads)s %(format)s'
 
 def getFFmpegThreads():
     if config.has_option('Server', 'ffmpeg_threads'):
diff --git a/plugins/video/transcode.py b/plugins/video/transcode.py
index 6586fe6..3b8724b 100644
--- a/plugins/video/transcode.py
+++ b/plugins/video/transcode.py
@@ -71,6 +71,7 @@ def transcode(isQuery, inFile, outFile, tsn=''):
                 'audio_codec': select_audiocodec(isQuery, inFile, tsn),
                 'audio_lang': select_audiolang(inFile, tsn),
                 'ffmpeg_pram': select_ffmpegprams(tsn),
+                'ffmpeg_threads': select_ffmpegthreads(),
                 'format': select_format(tsn)}
 
     if isQuery:
@@ -345,6 +346,9 @@ def select_maxvideobr(tsn):
 def select_buffsize(tsn):
     return '-bufsize ' + config.getBuffSize(tsn)
 
+def select_ffmpegthreads():
+    return '-threads ' + str(config.getFFmpegThreads())
+
 def select_ffmpegprams(tsn):
     params = config.getFFmpegPrams(tsn)
     if not params:
--
1.7.5.4

From f6f9a1900d7d5c54ef93a7b9863fbb7547e44376 Mon Sep 17 00:00:00 2001
From: Taylor Spencer <taylorspencer@hotmail.com>
Date: Wed, 11 Jan 2012 19:21:48 -0500
Subject: [PATCH] add select_ffmpegthreads() to ffmpeg settings in mp4_remux()
 function

---
 plugins/video/transcode.py |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/plugins/video/transcode.py b/plugins/video/transcode.py
index 3b8724b..acb8924 100644
--- a/plugins/video/transcode.py
+++ b/plugins/video/transcode.py
@@ -722,6 +722,7 @@ def mp4_remux(inFile, basename, tsn='', temp_share_path=''):
             'audio_codec': select_audiocodec(False, inFile, tsn, 'video/mp4'),
             'audio_lang': select_audiolang(inFile, tsn),
             'ffmpeg_pram': select_ffmpegprams(tsn),
+            'ffmpeg_threads': select_ffmpegthreads(),
             'format': '-f mp4'}
 
     cmd_string = config.getFFmpegTemplate(tsn) % settings
--
1.7.5.4

From bc2cacec494a26208b33fe1021c02b29c34996ae Mon Sep 17 00:00:00 2001
From: Taylor Spencer <taylorspencer@hotmail.com>
Date: Wed, 11 Jan 2012 20:52:24 -0500
Subject: [PATCH] convert getFFmpegThreads() return to str.  Removes having to
 convert to str in various other locations that call this
 function.

---
 config.py                  |    6 +++---
 plugins/video/transcode.py |    2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/config.py b/config.py
index 315defc..53abef9 100644
--- a/config.py
+++ b/config.py
@@ -287,14 +287,14 @@ def getFFmpegThreads():
        try:
           threads = max(int(float(config.get('Server', 'ffmpeg_threads'))), 1)
        except ValueError:
-          return 1
+          return ('1')
 
        #threads max is 16
        if threads <= 16:
-          return threads
+          return str(threads)
 
     #If option invalid or not specified, ffmpeg_threads should default to 1
-    return 1
+    return ('1')
 
 def getFFmpegPrams(tsn):
     return get_tsn('ffmpeg_pram', tsn, True)
diff --git a/plugins/video/transcode.py b/plugins/video/transcode.py
index acb8924..598e411 100644
--- a/plugins/video/transcode.py
+++ b/plugins/video/transcode.py
@@ -347,7 +347,7 @@ def select_buffsize(tsn):
     return '-bufsize ' + config.getBuffSize(tsn)
 
 def select_ffmpegthreads():
-    return '-threads ' + str(config.getFFmpegThreads())
+    return '-threads ' + config.getFFmpegThreads()
 
 def select_ffmpegprams(tsn):
     params = config.getFFmpegPrams(tsn)
--
1.7.5.4

From deb00b37581425ea2ef301bb47038a0752bc9aae Mon Sep 17 00:00:00 2001
From: Taylor Spencer <taylorspencer@hotmail.com>
Date: Wed, 11 Jan 2012 21:26:45 -0500
Subject: [PATCH] adds -threads variable to front end of FFmpeg command.  This
 has the affect of increasing multithreaded decoding
 performance during a file transcode.  Overall FPS may
 increase 10-15% while sending to the TiVo.  This may or may
 not have an affect on remuxing files as this largely is
 just a stream copy which relies on disk speed instead.

TODO: need to check for compatibility with older FFmpeg builds to ensure it doesn't break.  Tests OK with 0.8.x series.
---
 plugins/video/transcode.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/plugins/video/transcode.py b/plugins/video/transcode.py
index 598e411..50186fa 100644
--- a/plugins/video/transcode.py
+++ b/plugins/video/transcode.py
@@ -78,6 +78,7 @@ def transcode(isQuery, inFile, outFile, tsn=''):
         return settings
 
     ffmpeg_path = config.get_bin('ffmpeg')
+    ffmpeg_threads = config.getFFmpegThreads()
     cmd_string = config.getFFmpegTemplate(tsn) % settings
     fname = unicode(inFile, 'utf-8')
     if mswindows:
@@ -93,12 +94,12 @@ def transcode(isQuery, inFile, outFile, tsn=''):
             cmd = ''
             ffmpeg = tivodecode
         else:
-            cmd = [ffmpeg_path, '-i', '-'] + cmd_string.split()
+            cmd = [ffmpeg_path, '-threads ', ffmpeg_threads, '-i', '-'] + cmd_string.split()
             ffmpeg = subprocess.Popen(cmd, stdin=tivodecode.stdout,
                                       stdout=subprocess.PIPE,
                                       bufsize=(512 * 1024))
     else:
-        cmd = [ffmpeg_path, '-i', fname] + cmd_string.split()
+        cmd = [ffmpeg_path, '-threads', ffmpeg_threads, '-i', fname] + cmd_string.split()
         ffmpeg = subprocess.Popen(cmd, bufsize=(512 * 1024),
                                   stdout=subprocess.PIPE)
 
@@ -725,8 +726,10 @@ def mp4_remux(inFile, basename, tsn='', temp_share_path=''):
             'ffmpeg_threads': select_ffmpegthreads(),
             'format': '-f mp4'}
 
+    ffmpeg_threads = config.getFFmpegThreads()
+
     cmd_string = config.getFFmpegTemplate(tsn) % settings
-    cmd = [ffmpeg_path, '-i', fname] + cmd_string.split() + [oname]
+    cmd = [ffmpeg_path, '-threads', ffmpeg_threads, '-i', fname] + cmd_string.split() + [oname]
 
     debug('transcoding to tivo model ' + tsn[:3] + ' using ffmpeg command:')
     debug(' '.join(cmd))
--
1.7.5.4



FFmpeg decode multithread.diff.txt
 Description:

Download
 Filename:  FFmpeg decode multithread.diff.txt
 Filesize:  9.92 KB
 Downloaded:  22 Time(s)
Back to top
View user's profile Send private message
Iluvatar



Joined: 29 Feb 2008
Posts: 335
PostPosted: Wed Jan 18, 2012 5:16 pm    Post subject: Reply with quote
I have restructured this patch to be a little simpler, update error checking and to return None if setting is invalid or non-existent. This allows the FFmpeg thread setting to be included or not based on the check instead of defaulting to '-threads 1' when errors exist.

I still suck at this git thing so I couldn't figure out how to create a patch when compared to another branch such as wmcbrine. So here is the full code in plain text with a diff compared to my previous patch attached.

I would eventually like like to update the FFmpeg template to be completely inclusive of the global/input/output sequence instead of relying on manually hacking this every time someone wants to add a new option to the FFmpeg arguments. This would allow an easier template update instead for new FFmpeg developments.

config.py
Code:

def getFFmpegThreads():
    if config.has_option('Server', 'ffmpeg_threads'):
        logger = logging.getLogger('pyTivo.config')
        try:
            threads = config.get('Server', 'ffmpeg_threads')
            #older FFmpeg builds have history of crashing if threads < 1
            #threads max is 16
            if 1 <= int(threads) <= 16:
                    return threads

            else:
                logger.debug(threads + ' is an invalid ffmpeg_threads setting, must be between 1 and 16, using default')
                return None

        except ValueError:
            logger.debug(threads + ' is an invalid ffmpeg_threads setting, using defaults')
            return None

    else:
        return None


transcode.py
Code:

def select_ffmpegthreads():
    threads = config.getFFmpegThreads()

    if not threads:
        return ''

    return '-threads ' + threads


and in 3 locations (transcode, tivo decode, remux)
Code:

cmd = [ffmpeg_path] + select_ffmpegthreads().split() + ['-i',  .......... //the remaining code for cmd is the same depending on the function in which the line is located.


ffmpeg_threads_updated.diff.txt
 Description:

Download
 Filename:  ffmpeg_threads_updated.diff.txt
 Filesize:  4.14 KB
 Downloaded:  15 Time(s)

_________________
My pyTivo fork - Read link for changes
FFmpeg for OS X
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    pyTivo Discussion Forum Forum Index -> pyTivo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
Site is in NO WAY affiliated with TiVo Inc

Powered by phpBB © 2001, 2005 phpBB Group
phpBB SEO

Get pytivo at SourceForge.net. Fast, secure and Free Open Source software downloads

[ Time: 0.2119s ][ Queries: 19 (0.0303s) ][ GZIP on - Debug on ]