Hi all, I've attempted a different approach for bootstrapping Kotlin and I'd like to share with you the result. For the reminder, Kotlin is extremely difficult to build from sources because it is written itself in Kotlin and its build system is based on Gradle which also depends on Kotlin, that's the perfect scenario for a serious headache.
Instead of starting with a relatively recent release of Kotlin and a binary bootstrap compiler and standard library that are subsequently rebuilt, I tried to start further back in time with a release that doesn't depend on Kotlin and then build up a path toward the recent releases. TL;DR it didn't work, but I'd like to document the process if someone plans to follow this path in the future. The journey starts with locating the latest version of the Kotlin compiler that doesn't have .kt source files, and that's the version 0.6.786 (tagged as build-0.6.786). It goes back to October 2013. At this time Ant was the build system (the Ant build was dropped in the version build-1.2.30-dev-715 tagged in January 2018, so Gradle isn't an immediate concern). Java 6 was the version targeted by the built but I assume Java 7 (released in 2011) was used. For this bootstrapping process I'm using Java 8, which is the lowest version available in Debian unstable and the baseline for the current ant package. Kotlin 0.6.786 requires the IntelliJ SDK 132.27, the issue is the ancient SDK builds are not tagged in the Git repository. There are only branches for the stable releases (131, 133, 135...). Fortunately, the code on the branch 131 is good enough to build Kotlin 0.6.786. The branch 133 is fine too with a minor tweak (the class StubBuildingVisitor requires an extra constructor to preserve the backward compatibility with Kotlin < 0.6.1203). The SDK 133 can be used to build Kotlin up to the version 0.6.2122. A side note on the SDK: its dependencies are checked into the lib directory of the repository as binary jar files, and some of them were modified. We can replace them with Debian libraries, except one, ASM4, which was relocated to the org.jetbrains.asm4 package and got a few undocumented changes. Theoretically the lib/src directory contains the source archives for these modifications, but for ASM the sources don't compile and some modifications are missing (the jar was modified several times but the source archive was never updated). Without these modifications the Kotlin build fails with Java 8. After some investigation, it appears that the required modification is a change to the ClassReader class that ensures that parsing unsupported class file formats doesn't trigger an exception. This is a modification that's is still carried today by the Jetbrains variant of ASM (https://github.com/JetBrains/intellij-deps-asm/commit/db1b691). So Kotlin 0.6.786 + SDK 133 is able to build Kotlin up to the version 0.6.895. It fails on the version 0.6.900 due to a new annotation (jet.KotlinTraitImpl) added to Kotlin 0.6.855 and now used by the compiler. I opted to add the annotation to Kotlin 0.6.786 and continue. It works up to Kotlin 0.6.1364. With Kotlin 0.6.1379 the build fails due to the "externalannotations" attribute added to the <withKotlin> Ant task in Kotlin 0.6.1130 and first used in this version. So Kotlin 0.6.1364 now replaces 0.6.786 as the bootstrap compiler. Kotlin 0.6.1364 + SDK 133 builds Kotlin up to the version 0.6.1475 (note that 0.6.1379 fails due to the use of closure-compiler for the Kotlin to JavaScript compiler, I've simply disabled this part of the build). The version 0.6.1485 failed with an error I couldn't explain: [javac2] ERROR: /home/ebourg/kotlin/compiler/frontend/src/org/jetbrains/jet/lang/psi/psiUtil/jetPsiUtil.kt: (84, 58) Type mismatch: inferred type is (???) -> jet.Boolean but com.google.common.base.Predicate<org.jetbrains.jet.lang.psi.JetElement> was expected [javac2] ERROR: /home/ebourg/kotlin/compiler/frontend/src/org/jetbrains/jet/lang/psi/psiUtil/jetPsiUtil.kt: (84, 60) Cannot infer a type for this parameter. To specify it explicitly use the {(p : Type) => ...} notation It looks like the problematic function JetPsiUtil.getOutermostLastBlockElement() is neither used by the compiler nor the standard library, so I've removed it. The next failure is with Kotlin 0.6.1507 due to the new org.jetbrains.annotations.ReadOnly annotation which was moved to the runtime but used by the compiler before the runtime is built. Copying the annotation into the compiler source solves the issue. The next stop is Kotlin 0.6.1799 where the compilation breaks due to a bytecode verification error (java.lang.VerifyError: Bad type on operand stack). I suspect the bytecode verification got stricter in Java 8 and the bytecode generated by Kotlin back then is now considered invalid. I worked around the issue by adding the -noverify JVM argument (a trick shared by the Kotlin CLI script, so I assume it's ok). In Kotlin 0.6.1862 the Java to Kotlin converter (j2kConverter) fails to build, I'm not sure why but since it is optional I disabled it too. In Kotlin 0.6.1910 the following error appears: [javac2] ERROR: /home/ebourg/kotlin-from-scratch/kotlin.upstream/compiler/frontend/src/org/jetbrains/jet/lang/evaluate/OperationsMapGenerated.kt: (104, 64) Unresolved reference: compareTo I worked around this by commenting the offending line. The process continues up to Kotlin 0.6.2052 and this is the end of the road for Kotlin 0.6.1364 as the bootstrap compiler. The next version 0.6.2063 has unresolved references in compiler/builtins-serializer/src/run.kt. Kotlin 0.6.2052 becomes the new bootstrap compiler and we continue. Kotlin 0.6.2052 + SDK 133 builds Kotlin up to the version 0.6.2107. We switch again the bootstrap compiler. Kotlin 0.6.2107 + SDK 133 works up to Kotlin 0.6.2338. And now start the real troubles with Kotlin 0.6.2350. It breaks because this is the first version using the SDK 134, but there is no tag nor branch for the SDK 134. The 135 branch doesn't work due to the switch to ASM 5 (and Kotlin didn't switch to ASM 5 until the version 0.7.391). I've located where the version of the SDK changed from 134 and 135 (commit 4bb7296c2747fa73898247da03e7c9923935cc61) and I've built the SDK just before this point. The Kotlin build fails with this error: runtime: [echo] Cleaning /home/ebourg/kotlin-from-scratch/kotlin.upstream/dist/classes/runtime [mkdir] Created dir: /home/ebourg/kotlin-from-scratch/kotlin.upstream/dist/classes/runtime [java] exec() finished with INTERNAL_ERROR return codeEXCEPTION: java.lang.IllegalArgumentException: Missing extension point: com.intellij.psi.classFileDecompiler in area null [java] at com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.getExtensionPoint(ExtensionsAreaImpl.java:373) [java] at com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.getExtensionPoint(ExtensionsAreaImpl.java:37) [java] at com.intellij.openapi.extensions.Extensions.getExtensions(Extensions.java:110) [java] at com.intellij.openapi.extensions.Extensions.getExtensions(Extensions.java:98) [java] at com.intellij.openapi.extensions.ExtensionPointName.getExtensions(ExtensionPointName.java:48) [java] at com.intellij.psi.compiled.ClassFileDecompilers.find(ClassFileDecompilers.java:88) [java] at com.intellij.psi.ClassFileViewProviderFactory.createFileViewProvider(ClassFileViewProviderFactory.java:33) [java] at com.intellij.psi.impl.file.impl.FileManagerImpl.createFileViewProvider(FileManagerImpl.java:202) [java] at com.intellij.psi.impl.file.impl.FileManagerImpl.findViewProvider(FileManagerImpl.java:160) [java] at com.intellij.psi.impl.file.impl.FileManagerImpl.findFile(FileManagerImpl.java:328) [java] at com.intellij.psi.impl.PsiManagerImpl.findFile(PsiManagerImpl.java:188) I have no idea how to fix this. In a last attempt I tried to build Kotlin 0.7.391 with Kotlin 0.6.2338 + SDK 135 but it failed. This is where the story halts. To summarize: Step 1: build Kotlin 0.6.786 with a patched SDK 133 Step 2: build Kotlin 0.6.1364 Step 3: build Kotlin 0.6.2052 Step 4: build Kotlin 0.6.2107 (good up to Kotlin 0.6.2338). Step 5: how to build Kotlin 0.6.2350 ??? If someone figures out the next step, or has any hint about the last error, please ping me :) Emmanuel Bourg
