Making Linux talk to Windows and encode videos.
I run NextPVR on my Windows media computer / file server. It replaced the old Windows Media Centre when it got depracated in Windows 10. And it does a fine job of recording over the air broadcast digital TV (OTA DTv).
It records the raw, broadcasted video stream though, which results in a very large file on disk (even standard definition ends up being ~1GB / hour). So, I created a basic PowerShell script years ago to pick up the raw files and re-encode them with Handbrake into MP4s with aggressive compression.
A while back, I had to replace my media computer’s CPU and motherboard (due to my own incompetence). It previously has 4 cores, but now only has 2. Combine that with a few other CPU intensive tasks that run from time to time, and watching the recorded video in VLC can drop frames!
Run my video re-encoder on a new Debian Linux box with 4 cores against the raw video files on my media computer.
- Getting Debian talking to Windows via Samba.
- Porting my old Powershell script to .NET Core.
- Getting a recent version of Handbreak to run on Debian 9 Stretch.
- Making sure it Just Works™.
My first goal was to get my Debian box talking to my existing Windows file share of all my recorded TV. Ideally, this should be a permanent mount point via the SMB/CIFS file system.
I though this would be quite trivial, however it was more involved than expected. There were lots of resources available to configure Samba as a file server, domain controller, domain member, etc. But Samba’s own documentation was strangely silent on the topic of connecting to a someone else’s server.
Apparently, that was because I didn’t need to install Samba, but
Which is where the client part of Samba and SMB lives.
In the end, I installed both (because this computer would end up serving files via SMB anyway).
$ apt install samba cifs-utils
One important tool was
This is like an ftp client, but for Windows file shares, ie: SMB.
And, rather than figuring out the right syntax for mounting a file share correctly,
smbclient lets you quickly experiment and test.
Foolishly, I thought this would work:
$ apt install smbclient
I tried using different users, but there was little difference.
$ smbclient //loki.ligos.local/recordedTv -U murray
By now, I’d decided to get the Samba server working.
I hoped the smb.conf file might resolve these problems, but no luck.
I knew that Windows 10 disabled older versions of SMB (the protocol was radically re-worked in Windows Vista, and again in Windows 8), so I’d set my
smb.conf to contain the following:
smbclient doesn’t look at
So my additional settings were making no difference.
Instead, I needed to do this:
$ smbclient //loki.ligos.local/recordedTv -U murray -m SMB3
Now that I’d confirmed the details of connecting to Windows 10 from
smbclient, I was ready to mount the share more permanently.
I created a new folder ready for purpose:
$ mkdir /mnt/loki_RecordedTv
And looked into what I needed to tell
mount to actually make the mount happen.
Again, a little digging was needed in the mount.cifs man page to force the correct protocol version.
And I also wanted to save my credentials somewhere other than on a command line.
I ended up with something like this:
$ sudo mount -t cifs -o vers=3.0,credentials=/etc/samba/private/murray.credentials,sec=ntlmsspi //loki.ligos.local/RecordedTV /mnt/loki_RecordedTv
And the content of
And again, I had success!
$ ls /mnt/loki_RecordedTv
Years and years ago, Linux’s mount points were configured in
/etc/fstab - the file system table.
However, some searching on the ‘net showed me that
systemd was now capable of mounting things.
Well, as much as systemd always scares me, I might as well learn the new way.
However, a strange gotcha is that systemd enforces a naming convention of the unit name with respect to your mount point.
So for me,
/mnt/loki_RecordedTv must be a systemd unit called
Anything else will fail when you
systemctl enable wongly_named_loki_RecordedTv.mount.
I put my unit file in
You’ll notice the
noperm option which snuck in.
That tells the
cifs kernel module not to enforce unix permissions on the client side.
My mount point is owned by
root and has permissions
755, which ordinarily means only
root is allowed to write to it, no other user can.
noperm bypasses that.
Note, that this doesn’t bypass your Windows ACLs - they still apply, and if the user you specify in
mount doesn’t have write permission, then you’ll get errors.
All it means is you don’t need to get your unix permissions right - effectively, you’ve delegated all permissions checks to the file server.
OK, it’s time to make systemd do its thing:
$ systemctl daemon-reload
Well, it was successful until the network dropped out for a moment and the connection failed. Apparently, other people have the same problem.
Network failures happen even on wired ethernet networks from time to time. But I’ve also got my media server configured to shutdown from time to time (when there isn’t much to watch / record on TV). So I need to be confident the CIFS mounts will re-appear after any outage.
As with mount points, my ancient Unix wisdom said “use cron” (that is, edit
But remember, “
systemd can do everything”!
Systemd timers provide an alternative to
Arch Linux and Gentoo provided enough documentation and examples to get me up and running.
systemd timer service is more complex than a
cron job, but does allow slightly more flexibility.
First up, I made a script which did a
systemctl start for each of my mount points at
Then, I created and enabled a service unit as
This allows you to run the service on demand, without waiting for the timer.
/etc/systemd/system/smb-mounts.service I have:
As always, you need to enable the service, but then you can test it aside from any scheduling or timers.
status query will always show the service isn’t running; but that’s OK.
$ systemctl enable smb-mounts.service
Then I made a timer unit.
After my experience with systemd’s naming conventions, and every example saying you should name the timer
serviceName.timer, I went with
/etc/systemd/system/smb-mounts.timer, and everything was happy:
I’ve gone with a schedule of every 15 minutes.
But there are additional options under the
[Timer] section which let you define a particular time of the day / week / month, etc (similar to how
I went through the usual enable, start, and status to make sure everything is OK.
You can also use the
list-timers option to see all active systemd timers.
$ systemctl enable smb-mounts.timer
Finally, with a working network mount available, I can get down to my actual goal: re-encoding videos! Although my existing PowerShell script was working OK on Windows, I wasn’t quite ready to deploy it on PowerShell for Linux. Instead, I ported it to .NET Core.
I’d previously installed .NET Core, but the instructions are available from Microsoft.
The basic logic of this app is:
- For each
*.tsvideo file in my
- Check if the video has finished recording.
- If so, attempt to re-encode with Handbrake.
- If the re-encode was successful, delete the orginal
- Wait for a while.
- Goto 1.
The most exciting things to highlight are a) how to check if the video has finished recording, and b) how to detect a CTRL+C cancellation signal and react to it in short order.
Checking if the video has finished recording is done by attempting to open the file for exclusive read & write access. If the video is still recording, NextPVR will hold locks which prevent exclusive access and I get an exception.
private static (bool, string) CanOpenFile(string path)
Detecting cancellation is always a bit tricky. I long ago learned that the only way to respond to cancellation requests is co-operatively; you can’t just “kill” something and expect everything to be OK. That is, in my loop, I need to check for a flag representing that cancellation request, and if its set, you exit the loop.
bool cancelRequested = false;
That works fine, except the
Thread.Sleep() at the end.
If the cancellation event is received while I’m sleeping, I don’t actually do anything with that signal until the sleep has completed.
Which could be up to 15 minutes later.
Fortunately, we can use CancellationTokenSource to respond more quickly, while also maintaining the same
CancellationTokenSource cancelSignal = new CancellationTokenSource();
CancellationToken.WaitHandle.WaitOne() acts exactly the same as
Thread.Sleep() when you give it a timeout.
But, if cancellation is requested (via
CancellationTokenSource.Cancel()), it wakes up immediately.
Which gives the best of both worlds: waiting without burning CPU cycles, and responding promptly to a cancellation request.
If you’re interested, you can download the full source code for my little app.
Once the code was all done, I did a
dotnet publish -c Release deployed it into
/usr/local/bin/ReEncodeRecordedTv, and created a
systemd unit for it.
As this will run continually and poll the folder itself, I didn’t need a timer unit.
I could have removed the polling / sleep part of the program and let
systemd take care of that, but I’ve had issues in the past with task schedulers running multiple instances of the same task - and when your task consumes all available CPU for 5-60 minutes at a time, you really don’t want multiple instances!
Oh, and I also created a new system user
media-worker so it runs in a more isolated environment.
In particular, no
root privileges - which should mitigate a whole stack of potential security dramas.
My last problem was that the version of Handbrake available for Debian 9 Stretch via
apt was pretty old (0.10.2).
I’m a sucker for new and shiny, and you might as well install the newest version from the beginning (it will never magically get newer), so I wanted the current version (1.2.2).
Debian 10 Buster will have a more current version (1.2.2), but that doesn’t help me right now.
There were a few options:
- Try to use the version from Sid - that is, use a newer package than my version of Debian.
- Try to use the Ubuntu version of Handbrake on Debian (which is 1.2.2).
- Use the Flatpak version of Handbrake.
- Try to build from source.
Option 1 didn’t appeal to me at all: I had no idea if it would work or not, and I don’t have the skills with Debian pdkg to fix things when they break.
Option 2 was pretty much the same as option 1: I don’t know enough to be confident that it would work reliably.
Option 3 sounded more promising. Flatpak is a “it-just-works-everywhere” package manager for all manner of Unix operating systems. Except it needed to install ~300MB of stuff to do its thing for Handbrake. And, given Debian has the strongest array of packages of any Linux distribution (except perhaps Ubuntu), I didn’t like the idea of installing a whole new package manager just for Handbrake.
Option 4 had instructions to build Handbrake from source, plus a tarball for 1.2.2 available for download.
The dependencies for Debian were mostly straight forward, except a newer version of nasm was required.
Instructions were provided for getting
nasm from Debian Sid, but they were out of date and I needed to pick a newer package from here.
After that, a
./configure --launch-jobs=$(nproc) --launch --disable-gtk worked without a problem, taking around 10 minutes to build Handbrake CLI from source.
The result was a 40MB statically linked binary, which I copied to
$ ls /usr/local/bin/Hand* -alh
Once everything was in place, I watched one encoding run through (via
And then I left it overnight, and checked the next day that it was still working.
And left it until after a reboot, and checked all was well.
Each time, some minor corrections were needed.
It’s good to remind myself that whenever I implement something new, there will be teething issues which need to be resolved.
Usually they’re pretty minor (eg: I forgot to set the
mount systemd units to
enabled), but zero effort maintenance is a must!
So check and double check and then leave to bake, and then check again.
Only then did I say, it Just Works™.
That took much longer than I thought it should have! Oh well, I guess part of it is learning new things.
I’ve now got a separate encoder computer which uses Samba / CIFS to connect to my Windows media recording computer, checks for new recordings, and then re-encodes them to an mp4 with aggressive compression. Overall, the re-encoded files are between one third and one half smaller than the originals, so a pretty decent saving of disk space.
All that means, I can forget to delete stuff for twice as long before I get low disk space warnings! Yay for laziness!