 |
pyTivo Discussion Forum
Answers and the development of pyTivo a TiVo transcoding server
|
|
| Author |
Message |
Iluvatar
Joined: 29 Feb 2008
Posts: 335
|
 Posted: Thu Jan 12, 2012 7:49 pm Post subject: |
 |
|
| wmcbrine wrote: | My preferred solution for this would be more like: Have a reference counter for the temporary file; if the file exists at remux time, increment it; after a transfer, decrement the counter; if the counter is zero, remove the file. I didn't do it, because a) I was lazy, and b) how often would two TiVos request the same file at the same time, anyway? |
I understand. I'm just learning programming (starting with Python) so I suppose I wouldn't have though of that method.
| wmcbrine wrote: | But neither of these methods really accounts for crashes. Yours potentially leaves an ever-increasing number of partially-remuxed files, while mine could still leave one. |
My thought was to be non-destructive and to avoid the costly transcode(pull) at all costs. However rare it might be that a user already has a legitimate file existing with a .pytivo-temp extension I figured creating a new file was just easiest. Disk space is cheap and its not like they would be piling up infinitely. pyTivo crashing is really the only circumstance I have run across where the temp file is leftover. I have pyTivo auto restart on crashes and it was very annoying when my transfer is doing a full transcode instead of the faster remux because it came to a halt at the 'file exists' error. Cleaning up temp files all at once at my leisure is better to me than having to delete the temp file just to get the file to remux right now.
To be honest my preferred method would have been a config option that allows pyTivo to overwrite any file it comes across with a .pyTivo-temp ext. I figured that wouldn't go over well with anyone else though.
_________________
My pyTivo fork - Read link for changes
FFmpeg for OS X |
|
| Back to top |
|
 |
Iluvatar
Joined: 29 Feb 2008
Posts: 335
|
| Posted: Thu Jan 12, 2012 8:33 pm Post subject: |
|
|
| wmcbrine wrote: | My preferred solution for this would be more like: Have a reference counter for the temporary file; if the file exists at remux time, increment it; after a transfer, decrement the counter; if the counter is zero, remove the file. I didn't do it, because a) I was lazy, and b) how often would two TiVos request the same file at the same time, anyway?
But neither of these methods really accounts for crashes. Yours potentially leaves an ever-increasing number of partially-remuxed files, while mine could still leave one. |
Just wondering but assuming pyTivo crashes I am not seeing how your counter method wouldn't leave an ever-increasing number of orphaned temp files also.
Something like this: If the file exists already, counter is 1, create new temp file, counter is at 2, start transfer, pyTivo crashes or is killed. Counter or no counter there is now an additional temp file that didn't have a chance to finish transferring, decrement the counter and be removed.
Is this not the same (but a different way) as my method? Either way the temp file will always be removed, aside from pyTivo crashing. Am I thinking about this the wrong way?
_________________
My pyTivo fork - Read link for changes
FFmpeg for OS X |
|
| Back to top |
|
|
wmcbrine
Joined: 04 Jan 2008
Posts: 2009
Location: Maryland
|
| Posted: Fri Jan 13, 2012 10:55 am Post subject: |
|
|
Mine only makes one temp file per original file, no matter how many times it crashes. Yours makes up to 1000 temp files per original file. 
_________________
My pyTivo fork . My page |
|
| Back to top |
|
|
Iluvatar
Joined: 29 Feb 2008
Posts: 335
|
|
| Back to top |
|
|
Iluvatar
Joined: 29 Feb 2008
Posts: 335
|
| Posted: Sat Jan 14, 2012 12:52 am Post subject: |
|
|
Ok I have moved on to something else and have added a disk space check requirement before remuxing the temp file. This should prevent much wasted time if pyTivo tries to create a temp file and ends up running out of disk space before the remux can be completed. The outcome for not enough space being a debug message and completion of the transfer with a full transcode.
However before I post the patch I'd like to get some opinions on the best way to call for the check. I am not completely satisfied and it seems messy (it is fully functional though). This is based on the lucasnz fork which includes an extra temp_share_path var that I have to account for before passing to my check free space function. The wmcbrine fork doesn't have this and doesn't need the extra mess.
| Code: |
def push_one_file(self, f):
file_info = VideoDetails()
file_info['valid'] = transcode.supported_format(f['path'])
temp_share = config.get_server('temp_share', '')
temp_share_path = ''
if temp_share:
for name, data in config.getShares():
if temp_share == name:
temp_share_path = data.get('path')
#####
remux_path = temp_share_path
else:
remux_path = os.path.dirname(f['path'])
##### added a new remux_path var and the else statement
mime = 'video/mpeg'
if config.isHDtivo(f['tsn']):
for m in ['video/mp4', 'video/bif']:
if transcode.tivo_compatible(f['path'], f['tsn'], m)[0]:
mime = m
break
if (mime == 'video/mpeg' and
transcode.mp4_remuxable(f['path'], f['tsn']) and config.get_freeSpace(remux_path, f['path'])):
######### added the config.get_freeSpace check above
new_path = transcode.mp4_remux(f['path'], f['name'], f['tsn'], temp_share_path)
if new_path:
mime = 'video/mp4'
f['name'] = new_path
if temp_share_path:
ip = config.get_ip()
port = config.getPort()
container = quote(temp_share) + '/'
f['url'] = 'http://%s:%s/%s' % (ip, port, container)
|
I'd appreciate some thoughts to on how to more cleanly pass the proper output path to the get_freeSpace function.
_________________
My pyTivo fork - Read link for changes
FFmpeg for OS X |
|
| Back to top |
|
|
Iluvatar
Joined: 29 Feb 2008
Posts: 335
|
| Posted: Mon Jan 16, 2012 5:59 pm Post subject: |
|
|
OK well anyways here is the full patch. Based from lucasnz but could remove some code to rebase for wmcbrine since there is no temp_path setting.
It could use some prettifying but is fully functional. It saves me alot of time as my disk space fluctuates pretty drastically every few days. Usually my luck is that it writes out until just about the last byte of the video and then decides its out of space  .
| Code: |
From 8fca881775f087e03d9e79698e7af071f53ace78 Mon Sep 17 00:00:00 2001
From: Taylor Spencer <taylorspencer@hotmail.com>
Date: Mon, 16 Jan 2012 09:15:41 -0500
Subject: [PATCH] Check disk space before remux. This prevents wasting time
on remuxing a temp file without enough disk space to
contain the entire file. If check is False the transfer
will result in a pull(transcode) instead automatically.
Tested on OS X 10.7 and Windows 7 with Python 2.7.2.
---
config.py | 23 +++++++++++++++++++++++
plugins/video/video.py | 8 ++++++--
2 files changed, 29 insertions(+), 2 deletions(-)
diff --git a/config.py b/config.py
index fb9b017..6bc95ae 100644
--- a/config.py
+++ b/config.py
@@ -408,6 +408,29 @@ def get_tsn(name, tsn=None, raw=False):
def get_random():
return ''.join([random.choice(string.digits) for i in range(3)])
+def get_freeSpace(share, inFile):
+ logger = logging.getLogger('pyTivo.config')
+
+ # checks free space of given output path
+ if sys.platform=="win32":
+ import ctypes
+ freeSize = ctypes.c_ulonglong(0)
+ ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(share), None, None, ctypes.pointer(freeSize))
+ freeSize = freeSize.value
+
+ else:
+ s = os.statvfs(share)
+ freeSize = s.f_bavail * s.f_frsize
+
+ temp_fileSize = os.stat(inFile).st_size
+
+ # checks if enough free space exists on drive for temp file (plus padding)
+ if freeSize < temp_fileSize*1.1:
+ logger.error('Not enough disk space to remux')
+ return False
+
+ return True
+
# Parse a bitrate using the SI/IEEE suffix values as if by ffmpeg
# For example, 2K==2000, 2Ki==2048, 2MB==16000000, 2MiB==16777216
# Algorithm: http://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec/eval.c
diff --git a/plugins/video/video.py b/plugins/video/video.py
index 4865a5e..b8c16cd 100644
--- a/plugins/video/video.py
+++ b/plugins/video/video.py
@@ -425,7 +425,10 @@ class Video(Plugin):
for name, data in config.getShares():
if temp_share == name:
temp_share_path = data.get('path')
-
+ remux_path = temp_share_path
+ else:
+ remux_path = os.path.dirname(f['path'])
+
mime = 'video/mpeg'
if config.isHDtivo(f['tsn']):
for m in ['video/mp4', 'video/bif']:
@@ -434,7 +437,8 @@ class Video(Plugin):
break
if (mime == 'video/mpeg' and
- transcode.mp4_remuxable(f['path'], f['tsn'])):
+ transcode.mp4_remuxable(f['path'], f['tsn']) and config.get_freeSpace(remux_path, f['path'])):
+
new_path = transcode.mp4_remux(f['path'], f['name'], f['tsn'], temp_share_path)
if new_path:
mime = 'video/mp4'
--
1.7.5.4
|
| Description: |
|
Download |
| Filename: |
check_free.diff.txt |
| Filesize: |
2.8 KB |
| Downloaded: |
7 Time(s) |
_________________
My pyTivo fork - Read link for changes
FFmpeg for OS X |
|
| Back to top |
|
|
Iluvatar
Joined: 29 Feb 2008
Posts: 335
|
| Posted: Thu Mar 15, 2012 9:04 pm Post subject: |
|
|
wmcbrine I see you've been very busy lately with your additions. I'm glad that you decided to implement the audio transcoding for pushes with remux I'm sure it will be as useful for others as it has been for me. I've worked on ironing out some of the small kinks from lucasnz original implementation but as I've seen it there are only 1 or 2 useful fixes to be made (for now at least) that I'm still working on as time allows. One with the audio_lang function and stream selections and another when a AC3 track copy exceeds VOB specs when transcoding video but not audio.
_________________
My pyTivo fork - Read link for changes
FFmpeg for OS X |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
