Keeping the old (but supported) kernel in Ubuntu 20.04

[fox]

It's not like there's any rush, right? Even 19.10 is supported for 9 months so you could regress. You can still get newer versions of the programs you are specifically interested via snaps or repositories. Or get LM and you can continue using the 19.x version even when LM 20 comes out.

Yes there is a rush, partly because I'm doing a change involving 3 computers. The newest iMac came from school; no reason to have it there anymore. We have a very old iMac (20", 2009) in the "family computer" spot. It doesn't support the last four MacOS versions, although I have used a patch to allow it to update to the three year old version (MacOS High Sierra), which is still supported by Apple). I want to replace it with either the newest iMac that came from school or the one I am now using as a Linux box (2011 27" iMac) - my preference is the latter. The 2011 isn't having the same kernel problems as the newer one, probably because it's a lower dpi resolution (2560 x 1440). Its only problem with Linux is that the brightness controls don't work, but the brightness it's using is fine. I prefer the latest and greatest as my regular computer, but if it isn't secure, I'm better off to keep using the one I am now. Making any switch with the family computer is time-consuming, and I would hate to do it once and then have to repeat it once the Linux situation changes.

Staying with Ubuntu 19.10 isn't a reasonable option. First, it only gets three more months of support. Second, it doesn't run well on the 2015 iMac anyway (5.3 kernel), and the newest Ubuntu LTS runs as well or better than 19.10 on the 2011 iMac. Linux Mint 19.3 is a different, but unknown case. I'm assuming that the newer version will have some advances over 19.3, but I won't know until I try it. And I don't know if the new Mint version will continue to update the 4.15 kernel, or not. Yes I can run 19.3 perfectly on the 2015 iMac, but same goes for Ubuntu 18.04. But I would rather run Ubuntu 20.04 on the 2011 iMac than be forced to run 18.04 on the 2015.

Phew. Now I can take a breath!

Big problems? No. Maybe I have too much time on my hands. 

[Jason]

Staying with Ubuntu 19.10 isn't a reasonable option. First, it only gets three more months of support.

Ah, yes. I was thinking it was supported for 9 months but forget that it's already been 6 months.

[fox]

I have further information from a post I made on the Linux Mint forums. I asked about maintenance of the 4.15 kernel on Mint's soon-to-be-coming version 20, based on Ubuntu 20.04. The reply was "unlikely" because kernel 4.15 isn't in the Ubuntu 20.04 repositories. So I really need a solution to this problem before I consider upgrading to Mint 20, if I were to do so on my 2015 iMac.

The other thing I discovered is that Ubuntu has a software package site (here) where you can download packages in the respositories of all their main editions. I looked up the 4.15 kernel packages and can verify that they aren't in the 20.04 repositories. However they are in the 18.04 repositories, and can be downloaded as debs and installed. But here is the thing. There are 4.15.0-x generic versions where the x is numbered as high as 96. (The version I had in 20.04 was 72-generic.) Can I assume that the higher the number the more recent the security update, and that more will be added as the kernel continues to be patched? The reason I ask is because in my Ubuntu 18.04 partition, that number is only 37. So why aren't those higher numbers offered for my 18.04 installation when I check Software Update?

[Jason]

I'm not sure as to the answer to that question. Haven't trouble understanding all the stuff you're saying right now. Probably just been typing too much, here and on Facebook. I'l reply to it tomorrow when my mind is more clear. But I will say that the vanilla kernels at kernel.org (which are different than how the distros package them) show that there are several 4.x versions that continue to be supported and are considered long-term. The FAQ tells you what that term means.

You did ask one question before which I forgot to answer.

Too many questions and not enough answers. So here is another one for you. In my situation, would you run the 4.15 kernel if you knew it wasn't being updated?

It depends. I'd keep track of kernel releases through a support forum and see if the patches are for security or bug fixes or both. And if it's a security patch, look up the CVE related to it. CVE is a database of vulnerabilities. The CVE# is a specific vulnerability and they advise you what to do and how serious the vulnerability is. Some aren't serious or don't apply to your situation. For example, if your computer, being a desktop, is not accessible for people to log in from the internet, remote exploits don't matter because they have to be able to login before they can use it. But that's a lot of work to look up patch releases. But if you do a search for kernel updates for ubuntu, there should be release notes for each update that tell you what each patch is fixing and it will have a CVE# if it's patching a vulnerability.

But it still would surprise me that they wouldn't continue to support the kernel with patches since 4.x is still being updated. I'm using 18.04 on the server and it has 4.15.0-91-generic. It's still being updated and through Ubuntu. So why would they not give the option to use the previous kernel and have it patched? It's the same darn thing. But I'll look at the rest of this post tomorrow.

[fox]

Thanks, Jason. Very helpful, and I can now see a safe path to upgrading that doesn't look to involve a lot of extra work on my part. I look forward to your next post about this today.

Meanwhile, it looks like my bug report garnered some attention. I got an email today about it, asking me to test the newest upsteam kernel (5.7-rc2); it provides installation instructions. I'm going to try that this morning and report back.

Edit: 5.7-rc2 didn't work. It started out with the same slow boot and wouldn't let me past the login screen. When I put my password in and logged in, the screen just returned. I added these negative results to the bug report, and will hopefully hear back soon with another idea or more instructions to help them diagnose the bug. But I'm gratified that they are taking my report seriously.

Edit2: It was easy to find and install the 4.15 kernel modules that went missing, and I can now boot into that kernel again. My version is 4.15.0-72, and as Jason noted, 4.15.0-96 is available. I'll await advice as to whether I should upgrade to this, but in the meantime, I'm happy to know that I can, and that I will be able to securely operate with the 4.15 module until it is no longer supported in April 2023. But I'm still hoping for a fix of the current LTS kernel well before that.

[Jason]

Can I assume that the higher the number the more recent the security update, and that more will be added as the kernel continues to be patched? The reason I ask is because in my Ubuntu 18.04 partition, that number is only 37. So why aren't those higher numbers offered for my 18.04 installation when I check Software Update?

I think you already figured it out by now, but yes, the higher the number the newer the version. Not sure if it's a patch per se or just the distro version number. For example, I have version 5.3.0-46-generic on Kubuntu 19.10. The 46 part is the distro kernel release number. I figured that out from the official kernel pages which are where the original kernel developers hang out. All distros use their own kernel based on the kernels here. It says on this page under 'Distribution Kernels' that everything after the dash (i.e. -) means it's a distro kernel which I infer to mean that it's that number is from the distro kernel guys.

As to your second question, I guess you figured it out since you mentioned it in Edit 2 to your post before this. I didn't see the -96 release previously on the server when I checked it a few days ago so I was jealous until I logged into today and found it was available. So it must have just been released. On a server, I always apply the kernel updates. Since 18.04 is an LTS release, I believe it's tested well before it makes it into the repos.

Btw, what's curious is that I haven't upgraded to 20.04 yet but I see the new kernel already. It's version 5.3.0-46-generic. So maybe on the desktop for 19.10, you do have no choice, because I didn't explicitly choose to upgrade the kernel to the 5.3.0 series and it did. Oddly, at the kernel website, it doesn't list 4.15.x as a long-term release. This makes me think that in Ubuntu 18.04, Ubuntu is providing updates to it, not the kernel developers.

[fox]

....
Btw, what's curious is that I haven't upgraded to 20.04 yet but I see the new kernel already. It's version 5.3.0-46-generic. So maybe on the desktop for 19.10, you do have no choice, because I didn't explicitly choose to upgrade the kernel to the 5.3.0 series and it did. Oddly, at the kernel website, it doesn't list 4.15.x as a long-term release. This makes me think that in Ubuntu 18.04, Ubuntu is providing updates to it, not the kernel developers.

Actually, kernel 5.3 is the Ubuntu 19.10 kernel; 5.4 is the 20.04 LTS kernel.

I suspect you're right about Ubuntu providing these kernel updates. Is the "kernel website" you are referring to the Ubuntu kernel website or an upstream website?

[fox]

....
I didn't see the -96 release previously on the server when I checked it a few days ago so I was jealous until I logged into today and found it was available. So it must have just been released. On a server, I always apply the kernel updates. Since 18.04 is an LTS release, I believe it's tested well before it makes it into the repos.

When you say that you apply the kernel updates, do you mean that a program like Software Update tells you that they are available and you just let it install these? Or do you have to do what I did this morning, which is to hunt down the updates, download them and then install them with "dpkg -i"?

What is strange for me is that on my 18.04 partition, no new versions of the 4.15 kernel are shown as available when I check for updates. No -91, no -96. I can't check now, but I don't even think it showed the -72 available, which is what I carried over when I updated to 20.04.

[fox]

Update: Following instructions given by a dev responding to my kernel bug post, I have downloaded and tested various Ubuntu kernels from their Mainline website to determine the kernel version that started working poorly with my iMac. Turns out that the line was between kernels 5.2 and 5.3, with kernels 5.0, 5.1 and 5.2 working OK and kernels 5.3 and 5.4 not. He now wants me to test different stages of 5.2 to see if I can more precisely find the change where the problems started; he calls this a kernel bisect, and it involved testing kernels intermediate in sequence to the ones I have already found to be good and bad. Working this method, I was able to establish that all kernel versions of 5.2 were good until the rc1 for kernel 5.3. I'm hoping that this pinpoints the problem and generates a fix.

I'm guessing that one way to further pinpoint the problem would be to examine the changelog of the rc1 kernel, but I'll wait for further instructions.

[Jason]

Actually, kernel 5.3 is the Ubuntu 19.10 kernel; 5.4 is the 20.04 LTS kernel.

I suspect you're right about Ubuntu providing these kernel updates. Is the "kernel website" you are referring to the Ubuntu kernel website or an upstream website?

Thanks. I hadn't actually looked at the kernel version for 19.10 until your issue came up. The kernel website is the upstream website. It's the guys who make THE kernel. All distros take that kernel and then customize or add/remove things in it. You can take the kernel source code and make you own kernel by customizing the various settings and there are a lot of them. I used to do this way back more out of fun than any particular reason. The kernel website actually has several kernel releases doing at any one time which includes long-term versions, kind of like LTS.

[Jason]

Update: Following instructions given by a dev responding to my kernel bug post, I have downloaded and tested various Ubuntu kernels from their Mainline website to determine the kernel version that started working poorly with my iMac. Turns out that the line was between kernels 5.2 and 5.3, with kernels 5.0, 5.1 and 5.2 working OK and kernels 5.3 and 5.4 not. He now wants me to test different stages of 5.2 to see if I can more precisely find the change where the problems started; he calls this a kernel bisect, and it involved testing kernels intermediate in sequence to the ones I have already found to be good and bad. Working this method, I was able to establish that all kernel versions of 5.2 were good until the rc1 for kernel 5.3. I'm hoping that this pinpoints the problem and generates a fix.

I'm guessing that one way to further pinpoint the problem would be to examine the changelog of the rc1 kernel, but I'll wait for further instructions.

That's really cool that you're doing testing for them. It's one of the valuable ways to contribute to the Linux community. And, I'm just guessing here, there likely aren't a lot of people that run Linux on a Mac, if only because Macs make up a small slice of the over computer market. So your experience is likely invaluable to them. I certainly don't remember what it was a long time ago in a galaxy, well, in this galaxy, I submitted a bug report on SUSE back when I owned SUSE Desktop Professional and they fixed it. I felt pretty proud that I contributed by just finding a mistake. Filling out a bug report isn't always easy, either or having to do some specific testing the developer asks you to do.

I probably missed where you said it so pardon if this is a stupid question - what list did you submit the bug report to? Was it a Ubuntu list? Or were you on a kernel list like the ones on kernel.org? I only found out recently that most of the leading distros have people on their payroll that contribute to the kernel project which benefits all distros. A lot of the work involves supporting new hardware but also different filesystems, and some new features as well as fixing bugs and providing security patches.

[Jason]

When you say that you apply the kernel updates, do you mean that a program like Software Update tells you that they are available and you just let it install these? Or do you have to do what I did this morning, which is to hunt down the updates, download them and then install them with "dpkg -i"?

I use the following commands in the CLI, also known as the terminal:

Code Select Expand


sudo apt update
sudo apt full-upgrade

So, yes, I do what you said first except that I apply all updates. The first command in the terminal updates the catalogue from the repos and tells me what packages are available.  I don't pick and choose; I just install them all. I'm using an LTS version of Ubuntu so they've been well-tested or they wouldn't be there unless they're to fix new vulnerabilities. Even then, they are still using an older more stable version of the kernel and they're just fixing bugs and vulnerabilities, not adding new features, so it's much less likely to break something unlike with a desktop. Vulnerabilities are often exploited at the server level so it's important to stay up on those updates. The information on the servers is often more valuable - names, contact info, credit card numbers, banking information, passwords, etc.

[fox]

Unfortunately, this method wouldn't work for me, as the 4.15 kernel isn't in the Ubuntu 20.04 repos. The bug site I post on is a Ubuntu launchpad site: https://bugs.launchpad.net/ubuntu/+source/linux.

Additionally, I was asked to post a second bug report for the 5.7-rc2 kernel, which also slow booted, but this one wouldn't go past the login screen. (After login it would keep returning to the same login screen.) I hope I'm helping the Ubuntu developers with these reports.

[Jason]

Unfortunately, this method wouldn't work for me, as the 4.15 kernel isn't in the Ubuntu 20.04 repos. The bug site I post on is a Ubuntu launchpad site: https://bugs.launchpad.net/ubuntu/+source/linux.

I thought you were just wondering how I updated the kernel on the server, which is running on 18.04, an LTS release.

Do you have the link the post where you posted the bug report? I'm curious as to whether the guy who wanted you try stuff works for the kernel team generally or just the one that is used in ubuntu? It's possible Ubuntu to be paying someone to be on the upstream real kernel team, not just the ubuntu kernel team. This is confusing terminology. Let's refer to the kernel team as the guys who develop the kernel that all distros use a variation of and the kernel worked on for specific distro as abc kernel team (e.g. ubuntu kernel team). The actual kernel team won't provide support for distribution kernels or take bug reports from people using distro kernels.

Btw, I remember you once mentioning that you were worried about using the 4.15 kernel because you might not get updates for it. I read somewhere on an official Ubuntu page that the kernel is still updated but only for security issues rated with high severity. Bet you there is a way to use the 4.15 repos from the 18.04 LTS version and continually get that kernel updated until 2023. But you're going down a different road which is probably a better way to go about it.

[fox]

I don't know about the repos (I'm afraid to activate them), but I am able to get updates from here. However, they don't come automatically - I find out about an update through OMG! Ubuntu or some other site and then I go to the bionic kernel packages and download the four relevant files and install them with dpkg. An update was just recently released; the version is 4.15.0-101.

As for Mint, I'm still getting their 4.15 kernel updates automatically as a result of being on Mint 19. However, once I update that to Mint 20 when it comes out, the updates will no longer be automatic. I'll have to then look for an update source. Maybe it's the same as Ubuntu?