Linux Kernel Module Cheat

The perfect emulation setup to study and develop the Linux kernel v5.4.3, kernel modules, QEMU, gem5 and x86_64, ARMv7 and ARMv8 userland and baremetal assembly, ANSI C, C++ and POSIX. GDB step debug and KGDB just work. Powered by Buildroot and crosstool-NG. Highly automated. Thoroughly documented. Automated tests. "Tested" in an Ubuntu 20.04 host.完美的仿真设置,可用于研究和开发Linux内核v5.4.3,内核模块,QEMU,gem5和 C,C ++和POSIX。 GDB步骤调试和KGDB可以正常工作。 由Buildroot和crosstool-NG支持。 高度自动化。 彻底记录。 自动化测试。 在Ubuntu 19.10主机中经过“测试”。21世纪新政宣言(2020年4月5曰笫四次修改稿)(2020年6月19 很难产生最佳决策而大多是不优不劣成心对抗后的折衷方案使施政不理想选民失去信心;财团商界巨头对政治影响 并不只是几人而是几党几大群体竟争最高权力,强对抗易使社会撕裂更易使选民失望;另外还有竞选的形象口才和 不完全西化中西结合的,有利两岸和平统一的,最佳选择必是都相聚在中共也一惯崇敬的孙中山先生的旗帜下,完 倒退则是自取灭亡,必是遗臭万年的大罪人。中共改革派和大多数党员也不会愿意被往往只可能得势一时的倒退势 避免执政党自身权益高于一切的政党政治弊病。除了那拒绝政改变革而要倒退的是主动自杀,将不会有执政党下台 是和平自救或重整求生的唯一有利有效的办法。包括中共在内的几乎所有世人都认为中共政治体制改革已到尽头, (1)由新科举后全民大选出的行政府(简称官府),其各机构和职能与西式三权分立之行政相似;(2)民选不 都可在两岸各地发展并竞选各级议政府正副议长或议员。】7,理政府的正副理士、院士、考官、检查官和法官等 也有利于博才寡言的理论家和灵话善辩的活动家及两者兼优人才都有发辉和贡献的机会。这新政的民选与新科举不 易天下太平避免社会撕裂和政变动乱等,这三府合政体制该是真正理想而先进的东西结合的新型政体。【若中共签 2019.8.6签署:(在中共还未签署前或签署人及政党社团不具一定代表份量数额则暂不公开发表签署名) 同时发挥各自政治智慧尽全力劝说中共或其一切愿改革的派系人士签署本新政宣言。自我简介:男,70岁,政论 三朝罪恶元凶王沪宁:china-dictatorship-media-base: https://raw.githubusercontent.com/cirosantilli/chi https://raw.githubusercontent.com/cirosantilli/med -:sectanchors::sectlinks::sectnumlevels: 6:sectnums::toc: macro:toclevels: 6:toc-title:toc::[]== 【23】三朝罪恶元凶王沪宁大陆修宪香港恶法台湾武统朝鲜毁约美中冷战等都是王沪宁愚弄习思想极左命运共同 鼓吹新“长征”还要求用学习毛泽东著作与美打贸易战。引来美方冻结贪官境外资产断了外逃路,摆出与美国“决 操纵习近平的三朝罪恶元凶王沪宁不倒,不仅中国大陆无宁日,世界也面临中共新法西斯的威胁!(详请搜索郝雪 【22】习近平咋怕民主制而川普怎称他好友习小学毕业文革保送的"工农兵大学生"和在职读的"马列博士"全 【21】川普若与习签约落陷阱不挽救则连任无望香港恶法百万游行震惊世界,而中共仍未表中止恶法;仍未停反 【20】要求政府就两个问题向美国人民和国会作以检讨世界和平危害最大的中共大独裁者习近平的女儿在美国读 【19】短评一,刘昕仕途暗淡从刘昕几天停职准备和中宣部外交部及官媒造势来看,中美主播辩论原是要直播的 (大陆请别转贴答案)郝雪森原创2019.6.4(详请搜索郝雪森== 【18】纪念六四大陆全国"六月飞雪"活动倡议书我在大陆时有一邻居每年6月4日晚会在窗口点燃一烛,后来 【17】给习总女儿习明泽的公开信习明泽小姐:你比谁都清楚你父亲的学识和能力,能影响他决策的只有他身边 【16】若用这三张王牌中共必垮川普成为终结共产邪恶的英雄必赢得连任王沪宁愚弄习近平学毛极左倒退,使中 【15】王沪宁如此策划令习近平再活不过两年习实是小学毕业保送工农兵"大学"坐飞机,比毛泽东初中肆业还 【14】美中贸易战谁胜谁负,只需看中共是否仍然推行王沪宁极左路线美中贸易谈判过程中和履行协议期间,中 ,这时与之达成协议而不是加紧惩罚中共则是最大失败。美方要求中方结构性改革,中方只是书面承诺却行动仍向 ,有极左的中共世界必乱无经济秩序可言。眼下只需看中共是否有意釆纳其党内"平反六四"的意见,对六四学运 郝雪森== 【13】大陆军民今年要特别严防流血事件发生已是大陆政局最动荡的时期,很可能发生重大突发事件.当下王沪 祥请搜索: 郝雪森== 【12】已到中国变革复兴最佳时期,合力打倒三朝最恶元凶王沪宁!三朝罪恶元凶王沪宁一惯极左,其受益于小 中共必诱以川普家业和连任需要,但从不履行协议,必应借违约逐步打死中共,中共危境被动不会撕毁协议也无能 【11】打倒中共幕后操手王沪宁,六四和法轮功事件才有望翻案,朝核问题才有望解决镇压法轮功和最终为六四 2018年1月2日== 【10】告全国同胞书和致中共汪洋李克强等高官的公开信全国同胞们、汪洋李克强等中共高官们:美国总统川普 大造舆论,制造或寻找并参与终结中共的每一件力所能及的大事或小事,摧毁中共,复兴发展中西结合五权分立统 【9】中国新党筹建公告中华民族正处重大关头,大陆复兴中华民国的机遇来临,我们筹建“中国新议政党”简称 2018.7.30. [email protected](请转发,或区块链== 【8】 王沪宁愚弄习近平大倒退与美冷战必断送中共习近平承诺大开放,若真再开放将一发不可收拾,结局必是中共垮台 【7】 王沪宁加快倒退愚弄习近平走毛独裁路,决心与美冷战妄想成为世界霸主我有文章分析过,王操纵江泽民欺骗胡锦 【6】 大陆复兴中华民国全球华人行动起来,在国父孙中山旗织下统一中国!2018年将是中共内外交困全面走向崩溃 【5】总结百年号召百姓:三字今(经)我中华,数千年.饱沧桑,封建延.孙中山,有卓见.捣皇朝,民国建. 【4】习近平成为多线撑傀儡的政局预测作者:郝雪森皆知习近平小学毕业遇文革,初高中未读却保送工农兵大学 操纵玩弄多线撑傀儡习的有王沪宁刘鹤等“高参”。对刘等来说弄习出成绩有利仕途,而对王来说习无成绩下台才 【3】 若有下届则19大最大赢家王沪宁任总书记可能性最大作者:郝雪森我发表在北京之春网三篇之一《操纵江泽民欺 【2】 操纵江泽民欺骗胡锦涛玩弄习近平的王沪宁是在幕后最有实权的人作者:郝雪森{blank}[我去年开学时写 由此可见习近平的确是无知者无所畏惧,愚蠢者易被愚弄。王沪宁就是如此尽情如意地玩弄习近平,举最近一例, 重用无才无能但很听话的奴才。只不过,若说的好听,王沪宁很像王岐山比习近平江泽民等机灵得多,说的不好听 2017.10.16== 【1】 这位可敬的老奶奶教子可谓名留青史郝雪森----讲讲我哥第一次做小偷时听到总书记的母亲电话教训总书记的 为母亲筹到的医疗费全完了,哥心急如焚。返回打工住地时,哥遇上女同事的一个亲人,想到此人是在一家很富贵 他畏缩在茂密的枝叶中,最后决定等下一阵雷雨时逃走。此时,他还能清楚地听到那老奶奶的训话:“我不想听你 再加上你其实只有小学文凭, 你的能耐就一清二楚众所周知了。你刚进初一就文革停学,初中高中都没学过。后来保送清华上大学, 都知道那是可交白卷只是为了镀金的文凭。再后来你又当官在职读啥马列博士,国人谁会不知道这是假文凭?一个 你能离得开秘书半步?你完全被你周围的人利用和摆布,背离了你的父亲还蒙在鼓里, 你是活在他们编织的梦里!”老奶奶的这番话说的很激动也很不客气。我哥虽是打工仔,可也有作为一个大专毕业 他心想,东西是不能去偷,但能偷听到如此“国家大事”,没有白冒险一回,哥似乎是屏住呼吸倾听着:“想倒退 他说解放后二十多年里,毛泽东学斯大林给人民带来古今中外前所未有的苦难和浩劫. 毛所建立的体制必须改革,所以你爸在深圳搞了中国第一个改革试点。第二次是六四镇压学运以及苏联东欧共产党 要你们姐弟们远离政治,最好远离大陆,而你却没有做到。不过,你爸也理解你的苦衷,你四年工农兵大学是坐飞 工农兵大学除了学会26个英文字母,你也不认识几个英文单词。这也是你不能随前妻出国而离婚的原因之一,你 在这个体制内当官要牢牢记住, 1, 只能做改革派; 2,只能顺世潮顺民意做对民众有益的亊. 只有这样今后才可能不被清算,你爸临终时也只对你这件事很不放心”。“你身边周围那几个人,若不是只会阿谀 如此再进一步便是祸国殃民, 你必定遗臭万年,你周围的'高参'一个也逃不了,必遭严惩! 你职位最高,因此今后最大的野心家也只能出在你身边。他们只有让你学毛搞假民主只集中,独断独行,他们才有 要么学你父亲, 像个真正的男子汉, 大胆改革, 失败了也问心无愧。不行,回不了头就给我辞职. 你别无选择! 让别人或老百姓赶下台那就晚了,不仅会留下骂名,还可能落个前罗马尼亚的齐奥塞斯库的下场!一想到这点,我 智慧对任何人无论对庸人或能人来说都是有限的,能倾听大多数人的意见则是最大的智慧,对你来说是要多思而后 一位高龄老奶奶都能如此大义凛然, 年轻人能无动于衷吗?(有向这位高龄老奶奶致敬的读者请留言) 郝雪森 2016年9月10日# 第五个现代化:民主及其他魏京生序言现在报刊杂志和电台中不再震耳欲聋地宣传无产阶级专政和阶级斗争了。一 又称六四天安门事件 指1989年4月中旬开始的以悼念胡耀邦活动为导火索 由中国大陆高校学生在北京市天安门广场发起 持续近两个月的全境示威运动 6 7 也称八九民运 狭义上指六四清场 即1989年6月3日晚间至6月4日凌晨 中国人民解放军 武装警察部队和人民警察在北京天安门广场对示威集会进行的武力清场行动 8 9 1 11 六四天安门事件中华人民共和国民主运动及冷战的一部分Události na náměstí Tian an men Čína 1989 foto Jiří Tondl jpg在人民英雄纪念碑附近示威的学生日期1989年4月15日 1989年6月4日 51天 1个月2周又6天 地点 中华人民共和国 包含北京市在内的四百余个城市 起因胡耀邦逝世经济改革开放严重通货膨胀政治贪污腐败大量失业问题东欧剧变引发的世界民主化浪潮目标七项要 解决党和国家的贪腐问题 新闻自由与言论自由 追求社会平等 推动中国大陆政治民主化方法绝食 静坐 占领广场 设置路障 焚烧车辆结果国务院总理李鹏发布戒严令 宣布北京市戒严中国人民解放军进入天安门广场干预并驱散抗议群众 进行武力清场赵紫阳被罢免 民主改革派远离政治核心江泽民成为中共中央总书记 保守派获得提拔机会中华人民共和国政治体制改革停滞中华人民共和国加强对媒体的控制市场经济改革速度放缓中 转而形成对立局面冲突方中国共产党 中国共产党中华人民共和国 中华人民共和国政府执行机构 中国人民解放军 中国人民武装警察部队Police Badge P R China svg 中国人民警察Red star svg工人纠察队北京高校学生自治联合会等民间组织北京市和中国大陆各地大学院校学生部分工厂员工知识分子 1 领导人物强硬派 邓小平 中央军委主席 陈云 中顾委主任 李鹏 国务院总理 姚依林 国务院副总理 杨尚昆 国家主席 王震 国家副主席 李先念 全国政协主席 薄一波 中顾委副主任 李锡铭 中共北京市委书记 陈希同 北京市市长 刘华清 中央军委副秘书长 迟浩田 解放军总参谋长 江泽民 上海市委员会书记 温和派 赵紫阳 中共中央总书记 胡启立 中共中央书记处书记 万里 全国人大常委会委员长 彭冲 全国人大常委会副委员长 习仲勋 全国人大常委会副委员长 田纪云 国务院副总理 吴学谦 国务院副总理 徐勤先 陆军三十八集团军长 鲍彤 中共中央总书记政治秘书 阎明复 中共中央统战部部长 胡绩伟 全国人大常委会委员 李锐 中顾委委员 学生领袖 王丹吾尔开希刘刚柴玲周锋锁翟伟民张伯笠封从德李录沈彤王有才周勇军熊焱王超华马少方唐柏桥工人领袖 韩东方吕京花李旺阳知识分子 刘晓波2 1 年诺贝尔和平奖得主陈子明戴晴侯德健崔健江平方励之苏晓康陈佩斯茅于轼北岛包遵信汤一介辛灏年伤亡死亡18 至1 454 2 名平民不等 15至5 名军人及警察 3 各方估计并不相同 4 5 六四事件是中华人民共和国历史上的一个转折点 它的爆发标志着改革开放以来邓小平等人在中国大陆推动的后期政治体制改革失败 赵紫阳 鲍彤等中共改革派高层事后被撤职 而胡耀邦已在八六学潮中辞去中共总书记一职 于是198 年代所不同程度推动的自由化改革也就此停止 此后官方只批准了很少的游行活动 12 13 14 15 16 17 国际社会对此事件普遍表示了谴责和制裁 也有部分国家 多数位于中东及非洲 表示同情或者支持 而六四事件的后果除了造成政治从此转向收紧 经济影响也直接导致了中华人民共和国改革开放的放缓 直至1992年邓小平南巡后才重新提速 18 19 2 21 22 不过 邓小平任内推行的废除干部领导职务终身制则一直延续下来 期间更完成了3任政权的和平更替 直至习近平2 18年修宪后被废除 23 24 目录名称释义六四事件汉语六四事件字面意思六月四日发生的事件 显示 标音中华人民共和国政府使用的名称繁体字1989年春夏之交的政治風波简化字1989年春夏之交的政治风波 显示 标音汉语别称㈡繁体字八九民運简化字八九民运 显示 标音于香港维多利亚公园举办的维园六四21周年烛光晚会所摆设的标志历史名称广义上 “六四事件”或“六四天安门事件”是指1989年4月起于北京市发起并波及全国的抗议活动 更准确的称呼应为“八九民运”或“八九学运”等 事件的命名依据 一方面是要和过去发生在天安门广场的重要活动有一致的命名习惯 包括1919年的五四运动 1976年的四五运动等 有时候会直接简称“六四” 亦有人使用“六四运动”描述整起示威活动 与海外只集中在特写6月3日晚上清场的态度不同 在中国大陆境内使用“六四”这个词提及的范围与考虑的广度较大 25 指整个广义的“八九民运” 狭义上 “六四事件”是指中国人民解放军进驻天安门广场 要求抗议群众撤离的日子 尽管军队在6月3日晚上便执行清场任务 即“六四清场” 中国大陆以外的中文地区也将清场事件称作“六四镇压”或“六四屠杀” 26 官方称法自1989年以后 中国共产党和中华人民共和国政府也用数个名称指称六四事件 并被怀疑疑似借由修改事件称呼的方式 逐渐降低事件对往后社会大众的影响 27 在事件刚发生之际 中国政府将其命名为“动乱” 后升级为“反革命暴乱” 28 事件结束后以“六四风波”指称 后来在江泽民主政后期和胡锦涛主政时期 政府将当天的冲突全部改成更为中立的名称 也就是今日持续使用的“1989年春夏之交的政治风波” 29 3 或“1989年政治风波” 31 32 33 这类短语 27 34 2 19年习近平执政时期将“反革命暴乱”与“1989年春夏之交的政治风波”并行使用 35 西方称法西方世界在描述该事件的经过时 经常使用“1989年天安门广场抗议” 英语或“天安门镇压” 英语年代时西方新闻媒体经常使用“天安门大屠杀” 英语 Tiananmen Square Massacre 这类字词 36 但在近年的相关报导中则逐渐减少 37 主要是因为绝大部分暴力冲突并非发生在天安门广场上 而是在北京城西的木樨地 37 不过“天安门广场抗议”或是“天安门事件”等字词 容易让人误以为整个示威活动只发生在北京市 然而当时中国许多城市都有出现相关的抗议活动 37 代名词在中国大陆境内 上述名称皆于搜索引擎或公开论坛上被列为“敏感词” 为了要绕过网络审查 互联网上出现许多形容六四事件的替代称呼 38 包括有“5月35日” “VIIV” “6”和“4”的罗马数字写法 和“8平方” 82 64 等 39 4 “农历五月初一” 1989年6月4日为农历已巳年五月初一 这个表述一般情况很难被认为是六四的意思 对于“1989年” 则用“民国78年” “平成元年”等字眼规避审查 随着上述字词在中国网站上传播甚广 现在中国境内的多数网站也将这些视为自我审查词汇 在百度中搜索“天安门事件”则直接显示“四五运动”或者金水桥事件 但如果直接搜索“六四事件” 则会出现如中国网 中新网和中国日报等官媒发布的有关此次事件的资料 在百度贴吧里面搜索“六四事件”“5月35日”“8平方事件”“VIIV事件”都会显示“抱歉 根据相关法律法规和政策 相关结果不予展现” 事件背景改革开放参见 改革开放1977年7月 中共十届三中全会召开 大会通过恢复了邓小平的中共中央副主席 国务院副总理 中央军委副主席和解放军总参谋长的党政军职务 合称三副一长 邓小平正式复出 中共十一届三中全会在1978年12月召开后 获得最高权力的邓小平将改革开放列为重要政策 加速国民经济发展 41 同时邓小平提拔改革派成员担任重要的政府官员 其中中共中央秘书长胡耀邦在198 年2月被任命为中央书记处总书记 分割时任党主席华国锋的权力 同年9月赵紫阳则接替华国锋担任中华人民共和国国务院总理 华国锋在1981年下台后 胡耀邦接任中国共产党中央委员会主席 自此改革派进入中央最高领导层 尽管市场化的经济政策普遍受到人民的欢迎 但对官员腐败和裙带关系的担忧也不断增长 42 43 经济危机参见 价格闯关和198 年代末中国通货膨胀自195 年代以来 中国便长期透过中央制定的计划定价机制 让商品的价格稳定处在较低水平 但也减少了制造者增加产量的诱因 改革开放后 在经济改革初期 中国政府采取部分产品价格固定 部分商品允许价格波动的价格双轨制作法 但因市场上长期产品短缺而物价较高 部分人则可利用权力以低价购入产品 之后再以市场价格贩售 时谓“官倒” 44 此外 政府的货币供应量增发过多且过快 造成至少有三分之一的工厂无法获得利润 但1988年减少货币供应后 又使得许多金融贷款无法正常兑现 44 1988年 邓小平在北戴河会议上同意以市场经济为基础 让价格体系得以恢复正常 45 46 但价格管制将放松的消息传开后 随即引起民间恐慌 中国各地民众大量提领现金并购买商品囤积 45 不到两周内 政府便立即撤销价格改革的政策 但价格闯关带来的影响明显延续一段时间 民间社会面临快速通货膨胀的问题 在官方提出的消费者物价指数报告中 指出北京市的物价于1987年至1988年期间增长3 % 许多工薪阶层因为无法购买大众商品而感到恐慌 47 在新的市场经济体制下 许多无法获益的国有企业也被迫削减成本 让过去拥有工作保障与社会福利的铁饭碗开始面临生活的压力 47 48 社会问题改革开放后 中国社会出现了官倒 权钱交易 腐败 特权 贫富分化扩大等种种问题 22 49 5 51 此外 改革开放以后 改革派领导人设想知识分子会在往后发挥主导的功用 领导国家实施更多的经济改革政策 尽管政府陆续设立新的大学 并增加各校的招生名额 52 但情况并未如计划设想般实际发生 53 一方面因国家所指导的教育体制 并未充分和市场需求不断增长的农 轻工 服务业与外国投资等领域结合 54 另一方面因专精于社会科学和人文科学的学生 则必须进入有限的就业市场 52 新开设的私立企业并不接受国家分配毕业生 然而高收入的工作则由具裙带关系者取得 55 条件优厚的工作岗位都被取得后 剩下的职位往往是绩效较差的部门 掌握实质权力者则在该领域并无专长 47 面对惨淡的就业市场和有限的出国机会 知识分子与学生们认为凭借处理政治问题将能解决以上问题 这让北京市各个大学校园出现了研究政治为主的小规模“民主沙龙”社团 56 57 这些组织逐渐激发学生参与政治的兴趣 45 受到中国的经济社会逐渐朝向资本主义的影响 中国共产党名义上仍保留的社会主义 在意识形态上也面临信任危机 58 对于民营企业的审核制度 则让许多不良的商人能以宽松的法律优势致富 甚至常在过去强调“没有穷人”的社会中炫耀拥有的财富 47 59 财富分配不公的问题引起民众强烈的不满 也普遍对于国家的未来感到幻灭 6 派系斗争参见 中共八大元老保守派的中共元老中顾委主任中国国家主席李先念中顾委主任陈云 左 与国家主席李先念 右 当时人们希望中国政府能有其他改变的作为时 结果政府部门迟迟没有进一步的动作 58 在改革开放的政策制定和实施后 面对伴随而来浮现的种种问题 领导高层之间在处理办法上出现分歧 但尽管中国共产党内部因为意识形态而浮现派系冲突 双方人马都需要获得最高领导人邓小平的支持 才能实施各项重要决策 7 以中国共产党中央委员会总书记胡耀邦 中国国务院总理赵紫阳为首的改革派 又被称作“右派” 主张进一步实施政治自由化的方针 借由设立允许多种想法的渠道 让民众能够表达不满 并进一步支持改革 改革派成员还包括 胡启立 万里 彭冲 习仲勋 田纪云 鲍彤 阎明复 李锐 等等 61 62 另一方面 以中国共产党中央纪律检查委员会第一书记陈云 中国国家主席李先念为首的激进反改革派 又被称作“左派” 则认为改革开放已经施行过多政策 因而认为重新加强控制以确保社会稳定 并与中国共产党书面的社会主义主张一致 保守派成员还包括 王震 李鹏 薄一波 姚依林 邓力群 等等 62 政治体制改革主条目 中华人民共和国历史 § 政治体制改革198 年8月18日 邓小平在中共中央政治局扩大会议上作了 党和国家领导制度改革 的讲话 俗称“8 18讲话” 提出要进行政治体制改革 建议废除干部领导职务终身制 提倡民主集中制 并向全国人民代表大会提出全面修宪建议 63 64 1982年12月4日 第五届全国人民代表大会第五次会议审议通过了具有历史性意义的 八二宪法 该宪法也成为了中华人民共和国的第四部宪法 收入了许多宪政主义的内容和条款 为改革开放奠基 17 19 22 65 66 在邓小平的支持下 赵紫阳主持了后期政治体制改革随着改革开放的加速 中国社会出现了官倒 权钱交易 腐败 特权等种种问题 经济改革亦受到了原有政治体制的阻碍 22 49 5 51 1986年上半年 邓小平再次提出“政治改革”并启动了“政治体制改革”的研讨和制定 同年9月“中央政治体制改革研讨小组”成立 成员包括赵紫阳 胡启立 田纪云 薄一波 彭冲 49 65 67 68 69 1 月 赵紫阳提议的中央政改小组办公室成立 具体负责人包括鲍彤 严家其 贺光辉 周杰 7 邓小平的政治改革出发点是 在中国共产党一党专政的前提下 实行党政分开 提高行政效率 革除官僚主义弊端 推动经济制度进一步改革等 但不能照抄西方的宪政制度 他强调 “不能放弃专政 不能迁就要求民主化的情绪 要搞一个增强行政效能的体制 机构要精简 讲民主必须要和法制联系起来讲 把法制搞起来 才能有稳定的社会环境 我们的行政机构应该很有效能 ” 68 71 72 73 与此同时 其他人士还公开提出了“多党制” “三权分立” “议会民主” “司法独立”等西方宪政主义的架构 68 71 虽然这些与邓小平等人的中国共产党官方改革观点可能有所不同 但在当时比较宽松的政治气氛下 并没有受到过多的抑制与打压 68 1987年1 月 中国共产党第十三次全国代表大会在北京召开 邓小平主持了开幕式 赵紫阳作了题为 沿着有中国特色的社会主义道路前进 的报告 该报告由鲍彤负责起草 提出并论述了政治体制改革的方案和设想 阐述了社会主义初级阶段理论 提出了一个中心 两个基本点的概念 49 74 75 76 该报告的第五部分详细论述了政治体制改革 将邓小平198 年的“8 18讲话”作为改革的指导性文件 阐述了许多符合宪政主义的内容 其中包括进一步实行党政分开 权力下放 提倡法治和监督 完善选举制度等等 49 76 十三大还首次实行了差额选举 赵紫阳正式当选为中共中央总书记 鲍彤当选为中共中央委员 不久后鲍彤又被任命为中央常委政治秘书 49 74 1987年底 中共中央政治体制改革研究室成立 7 民间新思潮参见 第五个现代化 魏京生 资产阶级自由化 八六学潮和反对资产阶级自由化中国人民要现代化 首先必须实行民主 把中国的社会制度现代化 民主并不完全像苏联缔造者列宁编造的那样 仅仅是社会发达的结果 它不仅是生产力和生产关系发达到一定阶段的必然产物 也是生产力和生产关系在这个发达阶段以及更加发达的阶段中得以存在的条件 “”魏京生文化大革命结束后 早在1978年的拨乱反正时期 魏京生等中国知识分子便开始呼吁政治改革 并在北京市西单民主墙张贴不同政见的大字报 77 78 79 此时允许民众宣传政治自由和民主化的短暂时期 又被称作“北京之春” 但尔后魏京生在1979年3月遭到逮捕 8 西单民主墙也于同年12月时被迫封闭 81 1983年 中国共产党的保守派人士在发起了“清除精神污染”的左倾运动 1986年夏天 曾于普林斯顿大学任教的天体物理学教授方励之开始在中国各地大学展开个人访谈之旅 主要谈论的内容包括自由 人权 权力分立等内容 82 随后方励之成为当时社会大受欢迎的人物 83 他的发言记录也在学生间广为流传 84 对此邓小平曾警告方励之主张崇拜西方的生活方式 资本主义和多党制度 将意味着损害中国的传统价值观 社会主义的意识形态 以及中国共产党的领导能力 84 受到方励之的演讲 中国政治体制改革的重新开启以及世界各地爆发的群众运动影响 学生在1986年12月发起抗议活动 反对改革开放的步伐过于缓慢 其中参与示威游行的学生提出许多诉求 这包括有经济自由化 民主 法治等要求 85 虽然这次抗议最初是在合肥市附近进行 但很快地学生运动便蔓延至北京市等各大城市 对此中国共产党的中央领导阶层感到惊慌 并开始指责抗议学生试图煽动文革式的动乱 86 之后 中国共产党中央委员会总书记胡耀邦被中共内部指责对抗议活动的态度过于软弱 以及因为没有适当处理这次事件而引起社会动乱 胡耀邦遭到保守派人士大力谴责后 在1987年1月16日被迫辞去总书记的职务 但保留中共中央政治局委员的身份 86 87 88 在胡耀邦辞职后 中国共产党保守派在邓小平的支持下顺势展开了“反对资产阶级自由化”的左倾运动 开始针对支持胡耀邦观点 政治自由化和西方风格者进行打压 89 9 这项运动也制止了学生运动的发展 并且使得政治环境一度封闭起来 但胡耀邦也因而获得中国共产党党内的改革派人士 知识分子以及学生们的欢迎 91 92 但该运动此后遭到了代理中共中央总书记 时任中国国务院总理赵紫阳的反对 赵紫阳认为左派利用了反自由化运动来反对和否定改革开放 并以此说服了邓小平 该运动随后于1987年中期逐渐结束 67 93 94 95 国际局势主条目 东欧剧变和冷战事件起始与缓和胡耀邦逝世主条目 胡耀邦之死和对胡耀邦的纪念活动学生立起胡耀邦的巨幅画像 并在周围摆上花圈1989年4月15日 曾经在8 年代先后担任中共中央主席和中共中央总书记的胡耀邦因心脏病发作而逝世 随后引起学生强烈回响与悼念 并成为群众聚集的最初动力 96 97 大学校园里陆续出现许多歌颂胡耀邦的宣传海报 呼吁政府重新审视胡耀邦的观点 98 几天过后 大多数海报开始提到更加广泛的政治问题 包括有新闻自由 民主制度 以及官员贪污问题等 99 4月15日以后 一些悼念胡耀邦的民众也在天安门广场人民英雄纪念碑附近 自发组织小规模集会 同一天 北京大学与清华大学也在校园内设立胡耀邦的灵堂 北京当地学生陆陆续续聚集在天安门广场上 4月16日 位于西安市和上海市的学生也开始组织类似的小规模学生聚会 1 在部分大学生主导下 原本单纯悼念的活动转向要求政府控制通货膨胀 处理失业问题 解决官员贪腐 政府问责 新闻自由 民主政治与结社自由等 96 1 1 1 2 4月17日 中国政法大学的学生为了纪念胡耀邦而制做了大型花圈 在同一天有更多群众集结在天安门广场上 1 3 下午5时 5 名中国政法大学学生共同抵达靠近天安门广场的人民大会堂东门 表达哀悼胡耀邦之意 之后来自不同背景的演讲者举办公开演说 内容包括有纪念胡耀邦 讨论社会问题等 由于被视为将阻碍人民大会堂的运作 因此警方很快便介入示威群众的聚会 并试图说服学生离开天安门广场 4月17日晚上 3 多名北京大学学生在天安门广场进行学校学生的游行活动 很快地近千名来自清华大学的大学学生也参加游行 1 两队学生抵达天安门广场后 很快就与先前聚集在广场上的群众会合 随着活动规模的增长 聚会活动逐渐演变成为示威抗议 学生们开始向政府起草并提出7项要求 重新评价胡耀邦同志的功过是非 肯定其“民主 自由 宽松 和谐”的观点 严惩殴打学生和群众的凶手 要求有关责任者向受害者赔礼道歉 尽快公布新闻法 保障新闻自由 允许民间办报 要求国家领导干部向全国人民公开其本人及家属的实际财产收入 严查官倒 公布详情 要求国家有关领导人就教育政策的失误对全国人民作出正式检讨并追究责任 要求大幅度增加教育经费 提高知识分子待遇 重新评价反资产阶级自由化运动 并为在期间蒙受不白之冤的公民彻底平反 强烈要求新闻机构给予这次民主爱国运动以公正如实及时的报道 1 3 1 4 1 5 新华门事件示威学生曾一度聚集在中南海新华门静坐抗议 但最终遭到驱离4月18日上午 学生继续留在天安门广场 一些群众聚集在人民英雄纪念碑周围吟唱爱国歌曲 另外学生也在天安门广场上主办演讲活动 1 6 与此同时 数千名学生则聚集在中国共产党领导人居住的中南海入口新华门处 要求中国共产党的领导高层和学生之间展开对话 1 7 警方随即限制学生进入中南海内部 学生则决定原地静坐示威以表达不满 当天晚上 新华门前聚集了北大 人大 北师大 政法大学等校二三千名学生 围观群众六七千人 学生“会聚新华门是因为至今政府没有一个人出来表态” 学生多次齐声高呼“李鹏出来 ”“李鹏出来 ”的口号 并六次试图冲开警戒防线而未成功 1 8 1 9 11 许多学生认为他们遭到警方虐待 有关警察采取暴力驱离的传闻也迅速蔓延开来 111 新华门事件激怒了许多校园里的学生 许多过去没有积极参与政治事务的学生也因为这次事件 而决定加入抗议活动 112 在这段期间 一群自称“工人代表”的北京工人自治联合会则到处发布两份具挑战中央领导集团统治的传单 113 4月19日 立场靠近改革派的报纸 世界经济导报 决定出版纪念胡耀邦的专题报导 其中一篇由严家其所撰写的文章中 便对北京市学生发起的抗议活动给予正面评价 并且呼吁重新审视1987年要求胡耀邦下台的作为 不过在得知中央政府的立场渐趋保守后 江泽民要求 世界经济导报 删除相关的长篇敏感报导内容 但 世界经济导报 则以空白页刊登的方式抗议文字审查 114 最后江泽民马上解除总编辑钦本立的职务 115 其果断的行动赢得党内元老的正面评价 116 学运组成赵紫阳李鹏主张持续与学生进行沟通的赵紫阳 左 和主张对示威活动保持强硬态度的李鹏 右 由于胡耀邦曾经出任中共最高领导人的职务 中央决定为其举行国葬 仪式最后决定在4月22日举行 北京市人民政府下达命令封闭广场以举办葬礼 约有十万名学生则在前一天晚上无视命令 游行进驻至天安门广场 117 在4月22日当天 包括中央军委主席邓小平在内的中国党政领导高层皆前往人民大会堂内部参加典礼 并由中共中央总书记赵紫阳发表悼念词 中国国家主席杨尚昆主持仪式 尽管整个国葬过程向学生直接播出 然而由于纪念活动只持续了4 分钟便宣告结束 使得天安门广场的群众情绪更为高涨 7 118 119 12 虽然保安人员封锁了人民大会堂的东大门 但仍有数名学生共同突破封锁线 随后有三名学生跪在人民大会堂的阶梯上 表示要提交请愿书 并要求获得国务院总理李鹏的接见 121 然而没有任何中国共产党领导人自人民大会堂出现 这使得绝大部分学生感到失望与不满 121 122 4月21日至4月23日期间 学生们开始筹划成立真正的活动组织 1 在4月23日 北京高校学生自治联合会宣告成立 并选举当时就读中国政法大学的周勇军担任主席 而北京大学学生王丹 北京师范大学学生吾尔开希也被推举为各自学校的学生代表 随后北高联呼吁北京市的所有大专院校全面并无限期的罢课 以表抗议诉求 123 然而这样一个独立于管辖范围外的组织成立 挑战了中国共产党对学生的管理地位 124 另外一方面 位于湘潭市的湘潭大学学生也发起抗议行动 并且获得许多学校教授支持 同时武汉市当地的大学学生也组织起来 共同抗议湖北省人民政府 125 然而在4月22日黄昏 长沙市和西安市爆发了严重事故 其中包括在西安市有暴徒纵火毁坏车辆 房子 并且抢劫靠近西华门的商店 126 127 而在长沙市也有38家商店遭到暴徒抢劫 最后这两个城市共有超过35 人遭到了逮捕 128 随着国家局势变得更加动荡 中共中央总书记赵紫阳立即与中央政治局常委召开多次会议 对此赵紫阳强调要求学生停止进一步的抗议活动 而各自回到大学就读 他亦要求动用所有必要措施来解决动乱行为 而不同级别的政府应该与学生进行开放式对话 6 国务院总理李鹏则要求赵紫阳谴责示威群众 并认为应该要采取更加积极的防治措施 不过赵紫阳最后驳回了李鹏的看法 尽管中国共产党的领导高层就回应学生运动的方式意见分歧 而与赵紫阳关系密切的国务院副总理田纪云等人也建议赵紫阳继续留在北京市密切关注事态发展 1 但赵紫阳仍然依照原计划 应朝鲜劳动党总书记金日成的邀请 于4月23日飞往朝鲜进行国事访问 129 四二六社论主条目 必须旗帜鲜明地反对动乱为了抗议政府对 四二六社论 的定性 数百万名学生与群众发起了四二七游行 游行队伍举起写着“民主万岁 人民万岁 ” “廉洁的中国共产党万岁”的横幅赵紫阳前往朝鲜后 便由留在北京市的中共中央政治局常委 国务院总理李鹏代理领导党政机关 4月24日 李鹏和中共中央政治局委员兼北京市委书记李锡铭 以及国务委员兼北京市人民政府市长陈希同会面 希望能了解天安门广场上的情况 对此北京市官员想尽快解决危机 并认定抗议活动是一场阴谋 旨在推翻中国现有的政治制度 以及包括邓小平在内的主要党政领导人 在总书记赵紫阳缺席的情况下 中国共产党中央政治局常务委员会议认为必须立刻向示威群众采取态度坚决的行动 129 4月25日上午 中国国家主席杨尚昆和国务院总理李鹏前往邓小平的住处会见邓小平 邓小平同意政府采取强硬立场 邓小平还表示应该借由大众媒体适当地发布“警告” 借此抑制示威活动因为不断传播而扩大 13 这次会议成为中国共产党高层首次对抗议活动的正式评估 而重要问题的决定仍然以邓小平的意见为准 李鹏随后依照邓小平的意见下令起草一份公报 并向中国共产党各个机构和高阶官员要求应该设法对付示威群众 131 4月26日时 中国共产党的机关报 人民日报 头版发表社论 必须旗帜鲜明地反对动乱 四二六社论 指责“极少数别有用心的人”阴谋推翻中国共产党和现行的政治制度 132 133 134 然而这项声明激怒了学生 认为这是中国共产党故意要对付抗议活动 最后社论并没有令学生放弃示威活动 反而促成更多学生愿意团结 并共同表态支持学生活动 13 135 学生在天安门附近高举“学生的罪名 莫须有 ”的标语在北京高校学生自治联合会组织下 136 137 有五至十万名来自北京市各大学的学生在4月27日集结游行 经由街道前往天安门广场 138 学生团体成功通过警方设立的封锁线 并沿途接受以工厂工人团体为首的市民广泛支持 7 组织活动的学生领袖希望借由这次游行展现其爱国性质 特意淡化反对共产主义的口号 其中游行学生主要强调“反官僚 反贪腐 反任人唯亲”这一问题 不过学生仍强调会继续“拥护共产党” 135 这次示威游行迫使中国政府做出让步 同意与学生代表会面 4月29日 国务院发言人袁木会见由政府批准的学生社团代表 139 尽管会谈中讨论了包括报刊编辑 新华门事件 民主自由等广泛议题 并获得一些实质成果 然而包括吾尔开希等学生领袖则表态拒绝出席 14 141 142 五四对话4月3 日 中共中央总书记赵紫阳从朝鲜平壤返国并重新掌握党政权力 然而随着外界要求中国政府对学生示威活动的态度更为软化后 内部相关的讨论冲突反而更为加剧 以赵紫阳为首的温和派 主张继续与学生展开对话 以国务院总理李鹏为首的强硬派 则主张应该强硬地反对抗议活动继续进行 在5月1日召开的中共中央政治局常务委员会议上 赵紫阳和李鹏再度针对这一议题有所冲突 当中赵紫阳认为先前强硬派的作法已经证明并无实际的效力 因此政府特别允许这次活动才是唯一的选择 143 对于李鹏认为国家的稳定发展应该优先于任何事项 赵紫阳则反驳说中国共产党应该表态支持扩大民主和提升透明度的要求 最后在赵紫阳强力推行下 政府决定展开进一步的对话 144 赵紫阳随后开放新闻媒体积极报导抗议活动的发展 并在5月3日至5月4日期间发表了两次同情示威群众的演讲 145 赵紫阳发言中提到学生关切政府官员贪腐的问题是正当的 同时认为这次学生运动应该被视为一种爱国表现 146 144 在5月4日当天 有十万名学生在北京街头游行以纪念五四运动 147 同时再度重申先前示威游行所提出的要求 148 赵紫阳的发言实际基本上否定了4月26日 人民日报 发表的社论内容 149 这让很多大学生都满意政府所做的让步 15 5月4日结束时 除了北京大学和北京师范大学外 所有北京市的大学皆宣布罢课行动结束 随后大部分学生也逐渐失去参与抗议活动的兴趣 新闻自由获得口头保障 多数人主张以对话渐进推动民主 15 再度升级事件学生分歧与绝食1989年5月1 日 浦志强参加北京学运游行 要求“办报自由”和“结社自由”正当学生自治会所选举出来的正式对话代表团已经准备和中华人民共和国政府展 131 北京高校学生自治联合会组织领袖不愿意由正式对话代表团单方面控制整个抗议活动 151 在面对学生团体内部不和以及参与群众不断减少的情况下 包括王丹和吾尔开希等具有较大影响力的学生领袖要求采取更激进的作法来恢复抗议声势 其中他们认为中国政府所提出的“对话”只不过是一种诱骗学生就范的方式 因此自5月11日开始动员学生准备进行绝食 151 希望能够改变 四二六社论 的定性 152 最后学生决定在苏联共产党中央委员会总书记米哈伊尔 戈尔巴乔夫高调对华进行国事访问的前两天 自5月13日由柴玲宣读 绝食书 展开绝食抗议 153 154 其中学生领袖认为欢迎戈尔巴乔夫的仪式必定安排在天安门广场进行 因此借由绝食抗议便能作为筹码来迫使政府满足他们的要求 此外绝食获得社会大众广泛的同情 进而使得学生运动成为一种道德行为并且受到群众的追捧 155 而北京的抗议活动促使得其他城市的大学也陆续组织了抗议和罢课行动 同时有很多学生也纷纷前往北京市参加示威游行 其中在5月13日下午便约有3 万人聚集在天安门广场上 156 整体来说于天安门广场上进行的示威活动仍保有秩序 来自北京不同地区的大学学生每天发起游行以表达抗议要求并且表示团结 同时许多学生也会在行进过程中齐唱无产阶级国际主义运动著名的 国际歌 157 在5月中旬 学生发起绝食行动 促使中国各地四百多个城市陆续集结抗议 表态支持 158 苏共总书记访华主条目 1989年戈尔巴乔夫访华戈尔巴乔夫访问中国前夕 游行的学生在天安门广场拉起中俄双语横幅宣扬民主 图为“民主 我们共同的理想”标语1989年5月 时任苏共中央总书记的米哈伊尔 戈尔巴乔夫历史性访问中国苏联共产党中央委员会总书记米哈伊尔 戈尔巴乔夫是195 年代末中苏决裂后第一位正式访问中国的苏联领袖 两国关系恶化前苏共中央第一书记尼基塔 赫鲁晓夫曾于1959年访问中国 作为国宾出席庆祝中华人民共和国国庆1 周年大会 苏联领袖相隔3 年再次访问中国 象征两国关系改善 因此中国领导人非常重视这次国事访问 159 5月上旬至5月中旬时有关采访六四事件抗议群众的审查限制获得明显地开放 国家媒体开始播放包括绝食在内关于同情抗议群众的影像 然而由于担心示威活动将会失控 邓小平要求在苏共中央总书记戈尔巴乔夫访问中国期间应该清除广场上的抗议群众 为了达成邓小平的要求 赵紫阳决定仍使用柔性办法并且指示他的下属马上与学生进行谈判 155 赵紫阳相信此时仍能够成功借由爱国主义吸引学生的关切 并且让学生了解到如果在中苏首脑会议期间让其他人士知悉内部有动乱迹象的话会使得全国难堪 5月13日上午中共中央统战部部长阎明复召开紧急会议 16 并且邀请到重要的学生领袖以及包括刘晓波 陈子明以及王军涛等知识分子 161 阎明复说表示政府已经准备与学生代表展开直接对话 但前提是学生必须先撤离天安门广场以举办戈尔巴乔夫访问中国的欢迎仪式 这样也使学生领袖们之间陷入分歧 162 5月14日时 以戴晴为首的知识分子在中共中央政治局常委 中央书记处书记胡启立的许可之下直接通过政府审查在 光明日报 提出意见 呼吁学生应该要尽快离开天安门广场 但是许多学生却认为知识分子是为了政府发言而拒绝做出让步 156 当天晚上 以阎明复为首的中国政府代表团与担任学生代表的沈彤和项小吉展开正式谈判 其中阎明复肯定学生运动的爱国性质并且恳求学生从天安门广场上撤出 162 虽然阎明复的诚意成功促使得一些学生愿意达成妥协 但是随着不同派系的学生间无法事先进行协调或者提出连贯的要求而使得会议变得越来越混乱 不久学生领袖在得知政府并不愿意承诺公开直播问题的谈判过程后宣告会议无限期中止 163 之后阎明复直接前往天安门广场尝试劝离学生 甚至表示自己愿意被学生挟持以换取撤离的决定 然而学生之间并没有理会其劝告 7 而在隔天阎明复还向李鹏询问是否愿意应学生要求正式退回四二六社论的内容 并且将学生运动定调“爱国民主运动” 但这些建议都一一遭到李鹏的驳回 12 最后戈尔巴乔夫访问中国期间学生仍然决定继续留在天安门广场 也使得中国国家主席杨尚昆为戈尔巴乔夫访华举行的欢迎仪式上改在机场内进行 这次中苏首脑会晤于中苏交恶3 年后进行 除了标志中苏关系恢复正常外 同时也被视为中国领导人其具有重要历史意义的突破 164 然而相比之下由于学生仍然坚持在天安门广场上进行运动而为这次会谈带来尴尬 进而促使得许多原本偏向温和派的领导高层也开始转向愿意实施“强硬派”的作法 165 其中邓小平与戈尔巴乔夫在人民大会堂内举行两国领导人之间的高峰会时 学生群众则在附近天安门广场上发起示威活动 155 而在5月16日戈尔巴乔夫与赵紫阳会面后 赵紫阳则在国际新闻媒体前告诉戈尔巴乔夫表示邓小平在中国仍然是“至关重要的” 对此 邓小平认为赵紫阳的这句话是要将处理学生运动失当的过错归咎于他 166 这项言论标志着邓小平和赵紫阳两个中国最高层领导人之间决定性的分裂 155 局势升级1989年6月2日的天安门广场绝食抗议的作法很快便引起中国各地对于学生的支持和同情 167 并且在5月17日至5月18日期间数百万名居住于北京市的各行各业居民共同发起示威游行 而参与者还包括有中国人民解放军军人 警察人员 中国共产党党员或者是低阶的政府官员 168 同时许多中国共产党基层组织 中国共产主义青年团以及政府资助的工会也鼓励其成员公开参与游行活动 168 此外一些中国民主党派成员学生致信给李鹏以表达意见 而中国红十字会也特别下达通知并且安排大量人员前往天安门广场为绝食群众提供医疗服务 169 而在戈尔巴乔夫离开中国后 许多外国记者仍决定继续留在中国并且报导于首都北京市进行的抗议活动 这使得学生运动成为国际关注的焦点并且也让一些西方国家政府呼吁中国政府保持克制 17 171 至此原本于四月底声势衰退的抗议行动重新获得声望 5月17日时来自中国各地的学生陆陆续续涌进首都北京市以参与学生运动 而在中国各地四百多个城市也爆发规模不一的抗议活动 其中包括中共福建省委 中共湖北省委以及中共新疆维吾尔自治区党委机关甚至都遭到学生示威游行的影响 158 但是由于中国共产党领导高层迟迟没有针对北京发起的示威活动有明确的定位 这使得地方当局不知道如何处理当地的学生运动 而且因为示威活动合并了许多范围广泛且关注点不同的社会议题 这使得中国政府无法清楚分析哪些议题可以谈判 乃至于不清楚示威活动提出了哪些诉求 与此同时由于绝食抗议的行动其本身便具有“牺牲特质” 这使得无论是权威性还是合法性都因此而逐渐丧失的中国政府感到十分棘手 168 在种种因素所形成庞大的压力情况下 中国政府内部开始讨论将戒严作为一种应对示威活动的可行手段 172 5月18日 国务院总理李鹏在人民大会堂首次与学生代表会面 并且希望能够安抚受到大众关注的绝食行为 172 在会谈中学生领袖再次要求中国政府撤销 四二六社论 并且肯定学生运动为“爱国举动” 但对此李鹏则表示政府主要关切的是因为绝食而送往医院诊治的患者 尽管这次讨论仅取得了少数实质成果 但是学生领袖也因此得以在国家电视台重要节目上有了出现的机会 173 5月19日凌晨 赵紫阳则在中共中央办公厅主任温家宝陪同下前往天安门广场 而听闻消息陪同前往的李鹏则是抵达广场后马上离开 赵紫阳在凌晨4时5 分时借由扩音器直接呼吁学生结束绝食 并且告诉学生应该健康地活着 看到中国实现四个现代化的那一天 174 这是他最后一次公开露面 174 175 戒严清场主条目 六四戒严和六四清场戒严令您可以在维基文库中查找此百科条目的相关原始文献 国务院关于在北京市部分地区实行戒严的命令北京市人民政府令 1989年5月2 日 参见 中华人民共和国国务院令支持镇压的强硬派李鹏随着示威活动不断升级并且扩大 最后作为军方最高领导人的中央军委主席邓小平决定采取果断行动 一连串游行后 以中共中央军委主席邓小平及中国国务院总理李鹏为首的强硬派决定以武力解决示威 中央顾问委员会主任陈云 时任中国国家主席杨尚昆和前国家主席 时任全国政协主席李先念等多位保守派中共元老亦支持出兵 37 5月17日 政治局常委在邓小平的住所召开会议 12 176 在这次会议上赵紫阳不断让步的处理方针遭到了其他成员的批评 其中李鹏和邓小平宣称赵紫阳于5月4日发表的和解谈话使得学生不再惧怕中国政府 176 邓小平警告说如果北京市进行中的抗议活动不迅速平息的话 意味着中国将冒着经历另外一次内战或者是文化大革命的风险 而他的意见亦得到其他中国共产党党内元老的支持 177 邓小平随后表示应该宣布戒严以表达政府无法容忍抗议活动持续进行的立场 178 同时为了证明戒严有其作用而决定将示威群众描述为资产阶级自由化倡导者的“打手” 并且指称是幕后筹划的人士试图打击中国共产党的统治并且进一步实现他们个人的野心 179 同日傍晚中国共产党中央政治局常务委员会在中南海制定有关戒严之计划 期间赵紫阳表示由于无法实施戒严而准备辞去职务 18 同时他也不确定由中央政治局常委投票做出的戒严决定是否具有法律约束力 181 之后胡启立亦表示他并不愿意实施戒严 但相对的李鹏以及姚依林都表态支持宣布戒严的决定 乔石则提到虽然他反对政府再做出进一步的让步 但是他本人并不认为实施戒严为解决这一问题的有效方法 182 而出席此次会议元老人物中华人民共和国主席杨尚昆和中国共产党中央顾问委员会副主任薄一波则强烈要求中国共 之后担任中央军事委员会副主席和秘书长的杨尚昆更动用其权限开始调动军队进入首都北京市 182 5月19日 中国共产党中央政治局常务委员与军方领导人以及中国共产党党内元老会面 邓小平亲自主持会议并表示实施戒严是唯一的选择 在这次会议上邓小平宣布他“错误地”选择胡耀邦和赵紫阳担任他的继任者 并且决定从此将赵紫阳隔除在中国共产党高层领导会议外 邓小平还誓言要强硬处理赵紫阳的支持者 并且对此开始进行宣传工作 12 5月2 日 中国政府正式宣布实施戒严 183 184 132 并且从5个大军区中动员了至少3 个师的兵力 185 其中在中国人民解放军24个集团军中便至少有14个被要求部署军队 185 其中多达25 名士兵借由最终被送往首都北京市进行部署 其中有一部分军队则借由空运和铁路运输前往各自的目的地 186 而广州民航当局甚至还事先安排普通机票以准备随时运输部队 187 当天下午 杨尚昆当面明确北京军区司令员周衣冰为指挥 全权指挥戒严行动 188 然而中国人民解放军陆军部队进入城市后随即遭到大量集结的群众拦阻 在受到大量群众包围军车队伍并且阻止其进退的情况下使得部队在郊区无法继续前进 189 19 抗议群众也纷纷向士兵发表演讲并且呼吁后者加入他们的行动 同时示威群众还提供士兵食物 饮用水和相关用品 在部队迟迟无法向城市内部推进的情况下 中国政府于5月24日下令所有军队撤退至各个城市外的基地驻扎着 158 191 然而尽管示威群众成功逼使军事部队撤离被视为抗议活动“扭转颓势”的表现 但是中国政府仍然不断于中国各地调动部队以准备展开之后的行动 187 撤退与留守5月23日 天安门上的毛泽东肖像被泼墨与此同时学生运动的内部分裂则更为加剧 5月下旬学生所组织的抗议活动由于没有明确的领导人或一致的活动 情况变得越来越混乱 192 同时伴随着天安门广场上聚集著大量群众使得示威队伍出现严重的卫生问题 193 194 侯德健建议学生领袖进行公开选举以选出学生运动的发言人 但是遭到学生团体的反对 7 另外一方面王丹则认为近期中国政府将有可能发起军事行动以镇压示威活动 因此主张让学生先从天安门广场暂时撤回校园并且另外组成相关团体 但这个建议则遭到主张继续占领天安门广场的强硬派学生反对 随着派系冲突日益增加 各个派系开始争夺位在广场中央的学生广播中心 期望能够借由控制扩音器的方式掌握学生运动的控制权 各个派系也开始派遣一些学生前往火车站迎接来自全国各地声援的学生们 并趁机将他们拉到自己的派系之中以获得支持 7 学生团体开始指责其他派系的成员别有用心 这包括有勾结政府成员并且试图借由学生运动以获得个人成就 7 在5月27日时 香港将近三十万人则参与在跑马地马场举办的 民主歌声献中华 活动 不少香港名人应歌唱邀并且表示对北京学生的支持 隔天 在李柱铭 司徒华和其他组织的领导人领导下 香港15 万名群众聚集在于香港岛发起了大规模的抗议游行 而同一天世界各地也发起了全球华人大游行的活动 195 而在这期间 包括美国日本等政府也针对中国发出旅游警告 之后北京市的报刊上则陆续发表许多呼吁学生离开天安门广场并且结束抗学生运动的文章 其中在6月1日于 北京日报 刊载的 天安门广场啊 我为你哭泣 这篇文章中指称由于示威运动内部混乱和无序而使得作者感到失望 196 但是这些文章也使得许多不愿意离开天安门广场的学生感到愤怒并且开始组织抗议行动 196 数千名学生便列队自行游行至北京街头以表达不愿意撤离天安门广场 197 随后刘晓波 周舵与高新三名知识分子以及台湾歌手侯德健宣布发起第二次绝食活动 198 199 并且希望能够借此重新提振民主运动 2 而由于长期占领天安门广场后许多学生都渐渐感到疲累 这使得原先学生内部的温和派与强硬派之冲突也渐渐停息并且开始展开对话 2 1 之后刘晓波等人在发表的声明中提到绝食的目的是为了能够公开批评政府 同时提醒学生们他们现在的事业是值得奋斗的 并且促进学生能够继续占领天安门广场以提出继续改革的要求 2 2 6月2日晚间 一辆警方吉普车在行进时不慎撞击4名平民并且造成3人死亡 这件事造成示威群众开始担心军队和警察试图进驻天安门广场 2 3 对此学生领袖随即发出紧急命令 要求在主要的十字路口设置路障以防止部队进入城市中心 2 3 6月3日上午 学生和居民则发现有身穿便衣的军队试图携带武器进入城市 7 学生团体随即将其抓住并且把武器交还给北京市警方 2 4 学生随后于中南海的门口外进行抗议活动 但是遭到警方发射催泪瓦斯驱赶 2 5 另外一批没有携带武器的部队从人民大会堂出现后很快遭到抗议群众包围 并且在混乱中造成数人受伤 7 2 6 之后双方原地坐下并且开始吟唱歌曲 最后部队撤退回到人民大会堂大厅内 191 3日傍晚 中央电视台的新闻广播称戒严部队将会镇压动乱 并警告市民不要前往天安门广场 2 7 清场令指挥戒严行动的将领刘华清迟浩田戒严部队总指挥刘华清 左 上将和副指挥迟浩田 右 上将6月1日 李鹏向中共中央政治局提交 关于动乱的实质 报告 指称示威群众为恐怖分子和反革命分子 2 8 还指出抗议学生并不打算撤离天安门广场 同时示威活动也获得广泛支持 2 9 随后国安部也提交报告 强调资产阶级与自由主义已渗透到中国各处 西方观念给学生带来负面影响 21 国安部亦认为美军部队介入学生运动 期望借此推翻中国共产党的统治 211 这份报告在中国共产党党内成功营造出胁迫感 为之后的军事行动提供理由 21 同时中央政治局也收到戒严部队指挥部的报告 指出部队已经做好协助稳定首都现状的必要准备 种种因素让大多数中央政治局委员接受了戒严的必要性和合法性的说法 也同意之后借由武力清场以解决政治危机的方案 212 6月2日 随着学生的抗议运动有所增加 中共高层以武力解决政治危机的看法更加巩固 同日 中共高层再度召开会议 最终同意实施清场以“能够结束暴乱并且恢复首都秩序” 213 214 他们一致认为应尽可能和平地完成天安门广场的清场任务 但如果示威群众不愿配合的话 部队也被授权得以使用武力完成任务 国内报纸当天还报导军队部署于北京市十大重要关键地区 197 2 1 6月3日下午4时3 分 李鹏 乔石和姚依林3名政治局常委会见军方领导人 中共北京市委书记李锡铭 北京市长陈希同和国务院秘书长罗干 215 最后确定有关戒严实施的具体办法 213 216 会议确认将事件定性为“反革命暴乱” 必须果断采取强硬措施扭转局势 会议决定当日夜采取行动 “由周衣冰同志统一指挥解放军和武警部队力量 迅速开进天安门广场 坚决执行戒严任务” 217 在清场的当天晚上 中共领导人分别于人民大会堂和中南海监督执行状况 213 218 木樨地冲突6月3日晚间 国营电视台陆陆续续警告北京市居民留在室内 219 22 但受到前两周成功阻挡军队的激励 大批市民仍然走上街头以阻止部队行进 221 中国人民解放军部队从北京市各个方位逐步向天安门广场推进 分别由第38集团军 第63集团军和第28集团军负责西面 空降兵第15军 第2 集团军 第26集团军和第54集团军负责南面 第39集团军和卫戍第1师负责东面 以及第4 集团军和第64集团军负责北面 2 4 大约晚上1 时 第38集团军在广场西方约1 公里的长安街五棵松十字路口 开始向示威群众开枪 4 2 4 群众对于军队下令实弹射击感到惊讶外 转而开始向部队丢掷物品 2 4 当天晚上 32岁的航天技术人员宋晓明成为首位经证实的死者 2 4 之后军队遭指控使用射入人体会碎裂 进而造成严重创伤的达姆弹 158 222 1 时3 分 由于民众将双节无轨电车推到路上并放火焚烧 行进中的军队被迫暂时停在天安门广场西侧约5公里的木樨地 试图清除这些临时路障 223 224 住在附近公寓的居民亦出面试图拦阻军方车队 但第38集团军再度开火 并造成重大人员伤亡 158 1 5 218 根据天安门母亲运动调查后提出的死者报告中 共有36人在木樨地死亡 218 4 223 另外士兵还向木樨地附近的公寓开火 造成在建筑阳台或室内有人因而遭到枪杀 191 218 这包括数名在公寓观察事态发展的中国共产党高级党政官员 218 第38集团军最后以装甲运兵车将电车车厢撞开 并持续与尝试仓促搭建路障或组织人链的示威群众对峙 218 225 之后部队行经长安街经过南礼士路 复兴门 西单到天安门期间都有造成伤亡 2 4 226 而负责南面的空降兵第15军伞兵也使用实弹进行射击 并且在珠市口 天桥和前门等地也造成平民伤亡 4 但是亦有说法认为该类说法均为远距离观察导致的失真 在现场的CBS记者理查德 罗斯也称并未有士兵开枪 而仅仅试图驱散人群 这使得究竟是否发生开枪事故存疑 227 有说法称部队使用实弹进行射击并且造成死伤反而激怒北京市的居民 其中一些人开始以棍棒 石块和自制的汽油弹攻击士兵 228 甚至纵火焚烧军车 229 中国大陆当局以及其支持者表示军队主要是为了自我防卫而动用武力 并且提出部队的伤亡证明使用武器正当 而在报告中便有士兵在街上遭到活活烧死或者是被其他人殴打致死的纪录 23 而根据吴仁华的研究指出在6月3日晚上1 时军方开火后群众才开始向部队做出反击 231 232 不过在驱离过程中 亦有学生和居民试图保护遭攻击的士兵 而大多数的军事单位则拒绝向平民执行开枪命令 233 进驻天安门群众使用木板车运送伤员晚上8时3 分时 由于军队的直升机出现在天安门广场上空使得示威学生在各个大学校园呼吁学生加入其行列 晚上1 时 示威学生依照预定时间在民主女神的基座附近举办天安门民主大学的成立仪式 234 但是1 时16分时 由政府控制的扩音器警告说部队可以在实施戒严期间采取任何强制执行的措施 234 晚上1 时3 分 随着看见军队以实弹射击的目击者陆陆续续从北京市区西侧和南侧进入天安门广场 在天安门广场上的示威学生与群众也了解到有关暴力事件的消息 234 在午夜时分 学生扩音器则宣布一名学生在靠近中国人民革命军事博物馆的西长安街遭到杀害 这使得待在广场上的群众陷入忧郁情绪 234 学生指挥部副总指挥李录随即要求学生保持团结 并且坚持以非暴力的手段以争取继续占领天安门广场 234 凌晨 时3 分 吾尔开希则指控一名北京师范大学女学生在晚间离开校园后遭到杀害 之后吾尔开希便因为突发昏厥而被救护车带离广场 234 而在这时候 仍然有7 人至8 人继续留在天安门广场上 234 大约凌晨12时15分 军方开始发射照明弹以提供夜间照明 235 并且第一辆63式装甲运兵车自天安门广场西侧出现并且从广场前的道路快速通过 234 大约五分钟后出现第二辆装甲车 两者均往东长安街驶去 2 7 凌晨12时3 分左右则有2辆装甲运兵车抵达天安门广南侧 学生便陆陆续续向军方车辆丢掷水泥块 234 236 之后其中一辆装甲运兵车突然发生故障而无法移动 示威群众便用棍棒破坏车辆并且以浇上汽油的棉被放火焚烧 而因为车辆遭到焚烧而紧急逃出的3名军人则被示威群众殴打 但学生则组织警戒线并且护送3人到位于广场东侧中国国家博物馆的急救站医治 234 之后在承受极大压力的情况下学生领袖曾经一度放弃非暴力手段并且准备展开报复行动 237 其中柴玲便曾一度使用扩音器呼吁学生准备对抗“无耻的政府” 237 但最后她和李录同意继续维持和平手段的做法 并且没收学生所持有的棍棒 石块和玻璃瓶等可能被视为武器的物品 237 凌晨约1时3 分 第38集团军和空降兵第15军的队伍前沿分别到达天安门广场的南北两侧 238 他们开始封锁天安门广场四周并且将里面的示威学生和前往支援的居民分隔开离 而在过程中也杀死数名示威群众 37 同时第27集团军与第65集团军从天安门广场西侧的人民大会堂出现 而第24集团军也于东侧的中国国家博物馆开始进行部署 237 239 在遭到军队包围之后 数千名仍然留在广场上的示威学生与群众则开始往广场中央的人民英雄纪念碑聚集 24 凌晨2时后 部队开始尝试对人民英雄纪念碑旁的示威群众施加压力 241 而学生广播则不断呼吁军队放弃使用武力 并且提到 “我们是和平请愿 是为了祖国的民主自由 为了中华民族的富强 请你们顺从人民的意愿 不要对和平请愿的学生采取武力 238 ”大约凌晨2时3 分 几名工人则开始在人民英雄纪念碑架设其从装甲运兵车上拆解的机枪 并且发誓要向杀害许多示威群众的部队报仇 242 之后在侯德健劝说下工人们选择放弃武器 242 而刘晓波则在纪念碑的栏杆公开砸坏另外一枝没有子弹的步枪以重申非暴力运动的立场 242 群众撤离之后曾在木樨地看见军队枪杀民众的北京高校学生自治联合会常委邵江呼吁由知识分子带领示威学生与群 并且表示已经有太多人丧生 刘晓波原本表示不愿意撤离广场 但最后仍被说服并且和周舵 高新和侯德健一同与学生领袖商讨撤离问题 238 不过包括柴玲 李录和封从德等人在听到撤离意见后最初都拒绝撤离 238 凌晨3时3 分 在隶属于中国红十字会的两名医生建议下 侯德健和周舵同意先行尝试与士兵进行谈判 242 243 他们随即乘坐救护车抵达天安门广场东北角并且与第38集团军336军团政治委员季新国会面 24 季新国随即向戒严部队指挥总部转达请求并且获得同意为学生开辟往东南方安全撤离的通路 242 24 凌晨4点时天安门广场上的灯光突然熄灭 同时官方的扩音器宣布 “现在开始清场 同意同学们撤离广场的呼吁 238 244 ”不过此时学生们则是一同唱起 国际歌 245 并且认为军队准备执行最后一项清场任务 242 凌晨4点3 分时 天安门广场重新开启照明并发射一连串的红色信号弹 2 7 与此同时 部队开始从四面八方逼近纪念碑 132 246 随后部队在聚集于人民英雄纪念碑的示威群众周围1 米处重新部署 246 而侯德健回来后先是尝试说服事先知情的学生领袖接受他与部队的协议 在大约4时32分侯德健透过学生的广播表示他先行和部队达成谈判 246 然而许多第一次知道这次会谈的学生则气愤地指责他过于胆怯 246 封从德之后则在广播中解释由于已经没有时间召开紧急会议 而将以口头表决的方式决定示威学生之后的集体行动 但尽管“坚守”的声音比起“撤离”还要来得更加响亮 封从德仍然表示“撤离”意见较为多数而决定带领群众撤离天安门广场 223 不过大约在4时4 分时 穿着迷彩服的士兵冲向人民英雄纪念碑并且破坏学生的广播设施 246 247 而其他部队则殴打数十名在纪念碑旁的学生 并且扣押或者破坏他们的相机和录音设备 246 随后士兵开始强制驱散在人民英雄纪念碑附近的群众 之后也有学生和教授尝试说服仍然坚持坐在纪念碑底层的学生离开 246 大约早晨5时1 分学生们开始离开纪念碑 示威群众们手牵着手往广场东南角安排的通道撤离 132 223 248 不过由于当时坐在广场北部的学生颇多 因此有相当一部分学生是从广场北侧离去 246 2 7 而这时军方则要求那些拒绝离开天安门广场的学生必须加入撤离行列 247 除了以对空开枪的方式要求剩下的群众撤出广场外 246 249 并且还调动59式战车部队封锁前往广场的道路 25 251 而据从北侧撤离的学生所说 军队在纪念碑北面架起机枪向学生扫射 2 7 在确认所有示威群众都离开广场后 军方派遣军用直升机运送大型塑料袋并且命令士兵开始清理广场 251 北京医护人员对香港记者说 军队将广场的尸体装进塑料袋 由军用直升机运走 2 7 而6月4日早上6时 已经撤离天安门广场的学生队伍在沿着西长安街自行车车道上准备走回校园时 西单邮电局方向有3辆从天安门广场出发的坦克发射催泪弹并且冲撞人群 造成11名学生受伤 252 253 254 255 6月4日上午数千名先前撤离的示威群众 清场行动中伤亡者的父母以及被政府举动激怒的工人尝试从东长安街重新回到天安门广场 158 但是当人群靠近部队时士兵则随即往群众处开枪射击作为警告 158 256 然而由于有数人因而遭到部队枪击 之后暂时撤离该处的不满群众便又会尝试重新前往占领广场空地 158 257 之后群众多次尝试进入天安门广场内 但是军队则持续负责管理广场并且持续两周都不向大众开放 258 事件后续后续示威参见 王维林北京市区自6月4日军队已经控制天安门广场后便逐渐恢复稳定状态 不过在6月5日时西方媒体所拍摄和录制的王维林于长安街阻挡坦克行进的影像轰动世界各地 照片被视为整个六四事件标志性的照片之一 259 之后香港和澳门随即发起大规模示威游行以声援北京的示威群众 其中澳门有十多万至二十万人参与游行抗议 占当时澳门人口的一半 该次游行亦是澳门历史上规模最大的游行 26 另外也有一些国家也对于军队清场一事发起抗议活动 曾经加入天安门广场抗议活动的学生返回原本校园以及部队实施清场的消息传开后 中国内地包括成都市 西安市 武汉市 南京市 上海市和广州市等城市都在都爆发大规模的抗议行动并且持续数天 在广州 数万学生曾将主要干道海珠桥占领了四天 整个城市交通陷入瘫痪 到8日 因军队即将进城 人潮只得散去 261 而根据国际特赦组织的调查 在6月5日时成都市便至少有3 人丧生 其中成都市当地部队使用震撼手榴弹 警棍 刺刀和电击棒攻击平民 而当天晚上警方也刻意要求医院不能接受学生或者是提供救护车服务 262 而西安市在6月5日到6月6日期间 当地学生也发起大规模游行活动 并且联合参与罢工的工人开始设置路障 263 264 不过6月8日时陕西省人民政府便表示城市已经稳定下来 并且提出“先稳住动乱分子 尽量避免发生正面冲突 激化矛盾”的执行方针 265 上海则是在6月5日时由学生发起示威游行并且开始摆设路障 工厂工人亦发起大规模的罢工抗议行动 266 这些举动使得铁路与道路交通陷入瘫痪 同时早上许多大众交通工具也无法正常提供服务 266 英国广播公司则报导说 “数万名工作人士因而无法正常上下班 267 ”第二天在中共上海市委要求下 上海市人民政府派遣6 5 人清除道路上的路障以恢复通行 268 然而当天晚上8时3 分 来自北京的161次列车撞死在光新路道口5名尝试要封锁火车来往的群众 晚上1 时时现场便聚集超过3 万人并导致造成交通中断 而群众也开始殴打火车司机以及工作人员并且焚毁数辆火车车厢以表达不满 268 269 6月7日时 同济大学 华东师范大学和上海理工大学等学校学生纷纷强占各自学校的礼堂以及教学大楼 并且将其安排成灵堂以悼念六四事件的伤亡者 27 越来越多学生们参与架设路障阻碍交通的行动 但在听闻上海也有可能戒严后便有3 多人决定暂时离开校园 当天晚上 中共上海市委副书记兼市长朱镕基发表电视谈话 并且提到 “作为市长 在此郑重声明 市委 市政府从来没有考虑过要使用军队 从来没有打算实行军管或戒严 我们只追求稳定上海 稳定大局 坚持生产 保障生活 265 ”6月5日时 武汉当地约有2 名大学学生决定游行到天安门广场表达不满 271 另外示威群众也分别封锁武汉长江大桥的交通以及集结在武汉站前的广场 271 第二天学生继续在街头游行示威并使当地交通被迫中断 同时约有1 名学生决定直接在铁轨上静坐抗议 这导致来往北京 武汉到广州的铁路路线被迫中断 272 此外学生还鼓励当地各大大企业的工人发起罢工活动 272 6月7日凌晨学生们开始以公车和路障阻碍交通 并纷纷于各处架设灵堂来纪念六四事件的伤亡者 273 然而一小群激进的学生拦下一辆货运列车并且开始在列车上泼洒汽油 不过在准备焚烧时成功被警方阻止 273 但这使得当地警方与居民关系渐趋紧张 居民开始提领现金并且抢购物资 273 而6月5日 6月6日到6月7日 南京当地学生发起游行活动并四处发表演讲 此外示威学生亦封锁附近联络交通并尝试联合工人发起罢工 6月7日早上7时左右 包括河海大学在内几间大学4 多名学生以4辆公共汽车驾封锁南京长江大桥持续到傍晚 274 而南京大学为主的在校学生则在中央门各处摆放路障 274 一直到当天下午4时以后学生与群众才被说服并且撤离 274 也让交通一度恢复 274 然而6月8日 包括南京大学与河海大学等校学生重新控制了南京站周遭一公里的交通 275 同时学生们也陆陆续续于南京长江大桥上静坐表达不满 275 对于学生激烈的反应 中共江苏省委认为当地局势已渐趋失控 并陆续向学生表示公安部将会严惩行动的策划人 275 加强控制6月9日 在8 年代担任最高领导人的邓小平发表公开谈话 并批评示威活动者的目的军队在6月4日镇压天安门广场的示威群众后 中国政府开始加强控制新闻媒体和公民言论等自由 同时因六四事件惩处国内外媒体工作者 276 尽管国家媒体在军队实施清场刚发生后大多报导同情学生的内容 但是之后所有负责人遭到撤离职位作为处分 或者接受检讨和人人过关 其中在6月4日至6月5日在中国中央电视台播出的 新闻联播 上 个别负责报导此次事件因表现悲痛情绪的4名新闻主播分别是杜宪与张宏民 薛飞和李瑞英搭档 但事件之后央视过关学习 杜和薛的态度不改 却被调离处分三年后先后离职 而李张等人检查符合要求 才有了日后的天壤之别 中国国际广播电台英语部节目副主任 同时也是前中华人民共和国外交部部长吴学谦儿子的吴晓镛和英语播音员陈原能也因为其负责的节目对示威群众表 277 后来吴事后被捕 陈则被禁止出国 而 人民日报 由于发表同情示威群众的报导 包括社长钱李仁与总编辑谭文瑞等编辑都因而撤职 235 而包括吴学灿等编辑人员则是因为在未经许可的情况下出版特别相关报导 纷纷被判处有期徒刑4年 278 所有国际新闻媒体在北京市实施军事行动期间被勒令停止广播 同时中共当局早在5月24日就关闭了卫星传播的线路 279 然而许多广播公司仍然无视这些禁令而尝试借由试电话向外界报告情况 而许多相关的拍摄影片纪录也很快被偷偷地运出中国大陆 这也包括西班牙电视台在6月4日凌晨所拍摄的天安门广场情势独家纪录 28 而在军事行动期间一些外国记者则遭到中国大陆有关当局的骚扰 其中哥伦比亚广播公司的记者理查德 罗斯 Richard Roth 以及搭档摄影师便遭到中国大陆当局拘留 然而在他仍然不断透过移动电话报导天安门广场的情况 281 之后几名因为报导中国大陆当局派遣部队清场的外国记者在随后几个礼拜遭到驱逐出境 而其他记者则持续遭到中共当局骚扰或者是被列入黑名单之中 282 283 各国驻上海领事馆则被中国大陆当局告知并无法保证记者的人身安全 并开始要求每家新闻媒体必须遵从新颁布的准则内容 284 6月9日 中国共产党中央军事委员会主席邓小平在其他党政高层的陪同下前往中南海怀仁堂接见戒严部队高级干部 而这也是自从学生发起示威活动以来邓小平首次于公开场合出现 132 285 132 邓小平在之后演讲中称呼因为六四事件而丧生的解放军士兵为“烈士” 286 287 并且指出示威活动的目的是为了推翻共产党以及国家 希望能进一步“建立一个完全西方附庸化的资产阶级共和国” 288 邓小平认为示威者之所以不断强调包括官员贪污等等有关的投诉 便是为了掩盖其底下试图将当前社会主义制度加以取而代之的真正动机 289 他之后还以此观点表示 “这整个都是仍坚持帝国主义之西方世界的计划 他们企图让所有社会主义国家逐一放弃社会主义道路 然后将它们带往另一条充斥着国际资本以及垄断资本主义的道路 29 ”为了排除同情天安门广场示威群众的中国共产党党内人士 中国共产党领导高层发起了长达一年半的整顿方案以“严格处理内部强烈倾向资产阶级自由化道路的党员” 根据报导有将近4 名中国大陆官员被调查其在抗议期间的作为 有超过3 名中国共产党党员的职位因此被迫调动 甚至估计有超过1 名官员其政治可靠程度遭重新评估 291 而在这一期间 几名中国大陆外交官则前往国外要求提供政治庇护 292 而抗议行动造成中国大陆当局决定加强其作用 在六四事件发生后许多在198 年代引入的自由作法遭到撤销 同时中国共产党回到传统列宁主义的模式并且重新控制新闻出版和大众媒体 不过六四事件使得中国大陆当局了解到无论是中国人民解放军或者是北京警方都缺乏如橡皮子弹和催泪瓦斯等充足 因此在这次抗议行动结束后中国大陆各个城市的镇暴警察陆续获得非致命的相关装备 同时六四事件还促使得中国大陆当局决定增加国内“维稳”开支 并且扩大中国人民武装警察部队在镇压城市抗议活动时的权限 293 逮捕行动在所有被通缉学生领袖中排行第一位的王丹 同时他也是几位曾遭到中国大陆当局关押的学生领袖之一参见 黄雀行动和六四绿卡在军队成功控制天安门广场后 中国大陆当局开始针对参与示威抗议的群众展开大规模逮捕行动 并且也撤除相关工作或者支持抗议群众的政府官员的职务 294 根据北京市委办公厅所编出版的 1989北京制止动乱 平息反革命暴乱纪事 提到 北京市在“六四”后共抓捕了1 1 3名涉及“暴乱”的疑犯 有市民因为仅仅在街上拿走士兵遗下的军用包就被控以抢劫罪名 判处七年 十年的监禁 295 许多参与六四事件的人士之后陆续流亡海外 包括柴玲 吾尔开希等学生领袖便透过香港组织的黄雀行动 逃往美国 英国 法国以及其他西方国家 296 297 298 之后则被中国大陆当局长期禁止返回中国大陆境内 299 不过陈子明和王军涛则是在1989年年底准备流亡时遭到逮捕 中国大陆当局指控他们为整起抗议活动的“幕后黑手”并且于199 年判处13年有期徒刑 3 3 1 中国大陆有关当局陆续逮捕或者拘留了共计数万名来自中国大陆各地的群众 而根据对话基金会援引各省级人民政府的资料指出在1989年春天时有1 6 2名与抗议活动有关的人士被判处有期徒刑 3 2 许多人被安排至监狱监禁或者是送往劳改营 并且被禁止与他们的家人会面 而相关单位除了时常对这些涉嫌参与示威活动的群众以酷刑施压外 还将这些持有不同意见者与杀人犯或者强奸犯安排在同一个牢房内 同时由于许多人遭到逮捕使得牢房空间不足 甚至没有足够的空间让每个人都可以入睡 3 3 上海监狱和劳改队开展“平息反革命暴乱”为内容的思想政治教育 播放 飘扬 共和国的旗帜 等录像 3 4 截至2 12年5月为止 仍然至少有2名参与抗议活动人士仍然关押在北京市 另外还有5人则下落不明 3 2 另外曾经担任赵紫阳助手的鲍彤则被指控泄露国家机密和反革命宣传 并于1992年7月被判有期徒刑7年 3 5 王丹和赵长青则是少数几位被列为通缉犯并遭到逮捕的学生领袖 被列为学生领袖通缉名单之首的王丹先是被判处4年的有期徒刑 3 6 在1998年以保外就医的理由 离开监狱并获准移民前往美国居住 3 7 他在哈佛大学获得学士学位后 主要从事学术的工作 3 8 赵常青因为被视为在抗议活动中影响力较小的公众人物 在为期6个月的监禁后获得释放 但之后由于继续要求中共展开政治改革而再度遭关押 吾尔开希在事件后逃往台湾 并长期在中央广播电台从事政治评论员的工作 3 9 之后他曾三度表态愿意自首 31 并希望返回中国大陆探视亲人 不过最后都被遣返回台湾 311 李录则在华尔街成为投资银行家 并且成立了一家公司 3 8 柴玲先是流亡法国 之后再转到美国寻求政治庇护 后来她于美国高科技公司工作 并成立了关注中国大陆女性权益和计划生育问题的非营利组织女童之声 All Girls Allowed 3 8 312 2 8年11月3日至11月21日期间 禁止酷刑委员会举办第四十一届会议 并根据 联合国禁止酷刑公约 第19条审议会员国提交的报告 当中禁止酷刑委员会对于中国大陆当局处理六四事件的方式感到担忧 认为尽管许多家属提出“在1989年6月4日于北京镇压行动中遭杀害 逮捕或失踪”的申诉 但中国大陆当局却迟迟没有展开调查 313 同时禁止酷刑委员会也提到中国大陆当局并未告诉家属相关人士的下落 而对过度动用武力的负责人也无任何行政或者刑事处分 313 不过在2 9年12月 中国大陆当局回应禁止酷刑委员会建议时表示已经就“1989年春夏之交的政治风波”完全结案 314 并强调当时采取及时且果断的手段是必要和正确的 314 同时中国大陆当局认为将六四事件形容成“民主运动”已经扭曲了事件想要推翻共产党专政的本质 314 并且认为这些意见与禁止酷刑委员会的职责并不符合 314 死伤人数6月6日后的一天 北京街头一辆烧毁的车辆由于中国官方拒绝提供更多事件资料 导致数据模糊且出现多种版本 单是死亡人数的估计便从百余人至上万人都有 315 1989年6月6日 中华人民共和国国务院官员召开新闻发表会 发言人袁木表示“初步统计”包括部队士兵 大学学生 非法分子和误杀群众在内 有近3 人死亡 316 317 袁木还提到有5 名中国人民解放军的士官和士兵受伤 而包括违法暴徒和围观群众在内 有2 名平民受伤 316 318 根据北京市警方的调查 在北京市遭到杀害的平民“包括大学教授 技术人员 政府官员 工厂工人 小型私营企业拥有人 退休工人 高中学生和小学学生等 其中最年轻的仅有9岁” 319 而自中国政府于6月6日召开新闻发表会后 六四事件的实际死亡人数和天安门广场伤亡问题便不断出现争论 有“死者都在天安门广场外” 32 的说法 原因在于中国政府展开军事镇压后 便不断控制任何资讯的发布 之后则严格禁止在中国境内研究相关主题 使得今日对于实际死亡和受伤人数仍然不清楚 各方来源提供的伤亡估计亦有很大的出入 声称人数从数人丧生至数千人丧生的说法都有 5 根据 中华人民共和国国务院公报 1989年第11号 所提供资料 中国人民解放军军队 武警及警察有数十人被害 6 多人负伤 非军人有3 多人受伤 2 余人死亡 包括36名大学生 医护人员 群众 321 海外报道的死亡人数一般显著多于大陆报道 2 14年美国白宫解密文件显示约有1 454人死亡 4 人受伤 白宫的报告引述自戒严部队的消息人士提供的中南海内部文件 2 17年底 英国国家档案馆解密的文件显示 有中国国务院的成员 名字在档案中被涂黑 称1989年天安门事件至少造成1 名平民死亡 2 6月21日 纪思道在 纽约时报 专栏提到因为缺乏实物证据 而很难确认伤亡的实际人数 但也提到“合理数字应该是大约有5 名士兵或警察死亡 以及4 名至8 名平民丧生” 3 美国驻华大使李洁明则表示美国国务院外交官看见军队向未持有武器的群众开火 走访北京附近医院后认为至少有数百人遭到枪杀 322 而美国驻华大使馆的员工也曾前往现场搜集部队杀害民众的证据 323 子女因为这次事件丧生的丁子霖 张先玲等人则共同成立天安门母亲运动后 在中国政府阻止下持续尝试调查死者家属 并记录死者相关资讯 4 天安门母亲运动确认的死者清单从1999年提出的155人 2 5年提出的187人 2 1 年提出的195人 4 在2 11年8月共有2 2人获得确认 4 在天安门母亲运动所搜集的死者资料中 除了因事件而直接丧生的民众外 还包括4名自杀人士 以及6月4日后因抗议活动而遭杀害者 4 324 另一方面 根据吴仁华对于戒严执法行动的相关研究 只有15名军事人员经确认是因为遭到杀害而丧生 231 在已经确认死亡原因的15名军事人员中 有6名士兵是因为搭乘的卡车翻覆 车体随后燃烧而丧生 1 5 一名隶属第39集团军宣传单位的摄影师因为没有穿着制服 在拍照过程中遭到枪杀 以及在同年7月4日 一名第24集团军排长因为心脏麻痹逝世 231 剩下7名军事人员的死亡原因 吴仁华认为应该是排除天安门广场上的示威群众时 在任务过程中阵亡 231 此外 中国各地仍有示威者伤亡 由于当地记者较少 伤亡程度难以估计 而根据国际特赦组织的调查 在6月5日时成都市便至少有3 人丧生 其中成都市当地部队使用震撼手榴弹 警棍 刺刀和电击棒攻击平民 而当天晚上警方也刻意要求医院不能接受学生或者是提供救护车服务 间接导致死者增加 325 位于成都的天府广场因此也被称为成都“天安门” 326 地点争议香港在2 1 年举办的平反六四大游行所使用的抗议牌子至今大多认为绝大部分的枪击事件 发生在天安门广场以外处 另外也有报导指称学生离开天安门广场后 军队仍在北京音乐厅附近向学生开火 3 而中国政府在6月6日召开的新闻发表会上 军事发言人张工坚持表示并无民众于天安门广场上遭到枪杀 军队亦没有使用坦克辗压在广场的民众 316 318 邓小平在6月9日的谈话中提到 “让大家看看 解放军究竟是什么人 有没有血洗天安门 流血的到底是谁 这个问题清楚了 就使我们取得了主动 286 ”清场期间负责监督执行状况的戒严部队副指挥迟浩田 则在1996年访问美国时坚持强调天安门广场上并没有人死亡 327 相比之下 西方的新闻媒体则广泛报导天安门广场上的“屠杀” 328 柴玲便表示坦克进入天安门广场后辗压帐篷 并杀害坚持不肯离去的学生 一些从广场撤离的学生则认为 有群众因待在人民英雄纪念碑附近而遭到杀害 329 之后军方出动直升机清理天安门广场 更让怀疑论者认为中国政府试图掩盖广场发生屠杀的事实 251 但侯德健 潘文等同样在待在纪念碑旁的人士 纷纷表示尽管在天安门广场有听到枪声 但并没有看到任何针对示威群众的大规模持枪射击 或是坦克辗压群众的情况发生 33 美国国务院内部针对事件提出总结时 则提到 “屠杀发生在长安街等北京主要干道 而不是在天安门广场本身 331 ”这些说法使得之后记者依照现有证据进行判断 认定6月3日晚上至6月4日凌晨并没有民众于天安门广场上丧生 332 在维基解密所泄露的外交电报中 6月3日晚上至6月4日凌晨待在天安门广场东北角的智利外交官卡洛斯 加洛 Carlos Gallo 便告知美国大使馆工作人员在广场上并未目击到群众遭遇枪击 在这期间只有听到零星的枪声 251 不过另一篇外交电报提到一名执行清场任务士兵的母亲 从自己儿子口中得知当时隶属的部队安排在天安门广场东南角 并曾使用机枪扫射而造成许多平民丧生 该电文还提到这名士兵为基督徒 333 尽管电报中提到无法验证来源是否真实 但仍然被美国驻上海总领事馆当作目击者报告看待 333 各方反应主条目 六四事件反应官方定调这场风波迟早要来 这是国际的大气候和中国自己的小气候所决定了的 是一定要来的 是不以人们的意志为转移的 只不过是迟早的问题 大小的问题 而现在来 对我们比较有利 最有利的是 我们有一大批老同志健在 他们经历的风波多 懂得事情的利害关系 他们是支持对暴乱采取坚决行动的 虽然有一些同志一时还不理解 但最终是会理解的 会支持中央这个决定的 事情一爆发出来 就很明确 他们的根本口号主要是两个 一是要打倒共产党 一是要推翻社会
Alternatives To Linux Kernel Module Cheat
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Cirros100
6 days ago17Shell
Mylinux70
3 months agogpl-2.0Shell
myLinux is an embedded operating system based on Buildroot and Finit
Tflite_zynq37
6 years ago1gpl-3.0C++
Example code and instructions on getting Tensorflow Lite running on a Xilinx Zynq
Qemu_devices23
7 years agoC
PCI device for qemu with mmio, pio, dma
Meson Tools18
4 years ago3gpl-2.0C
Tools for Amlogic Meson ARM platforms
Camkes Vm Examples12
4 months ago13otherC
Sdk_48111
7 years agoC++
Toolchains6
5 years ago
Buildroot toolchains for all archs (arm, mips, x86, mipseb)
Inphic S8054
8 years ago
Teardown of Inphic S805
Arm_emulator4
2 years agogpl-2.0C
A simple armv4 emulator with embedded freertos and linux operating system support, Demo:
Alternatives To Linux Kernel Module Cheat
Select To Compare


Alternative Project Comparisons
Readme

Linux Kernel Module Cheat

64534859

The perfect emulation setup to study and develop the Linux kernel v5.9.2, kernel modules, QEMU, gem5 and x86_64, ARMv7 and ARMv8 userland and baremetal assembly, ANSI C, C++ and POSIX. GDB step debug and KGDB just work. Powered by Buildroot and crosstool-NG. Highly automated. Thoroughly documented. Automated tests. "Tested" in an Ubuntu 20.04 host.

The source code for this page is located at: cirosantilli/linux-kernel-module-cheat. Due to a GitHub limitation, this README is too long and not fully rendered on github.com, so either use:

Xinjiang prisoners sitting identified

1. --china

The most important functionality of this repository is the --china option, sample usage:

./setup
./run --china > index.html
firefox index.html

The secondary systems programming functionality is described on the sections below starting from Getting started.

Tiananmen cute girls

2. Getting started

Each child section describes a possible different setup for this repo.

If you don’t know which one to go for, start with QEMU Buildroot setup getting started.

Design goals of this project are documented at: [design-goals].

2.1. Should you waste your life with systems programming?

Being the hardcore person who fully understands an important complex system such as a computer, it does have a nice ring to it doesn’t it?

But before you dedicate your life to this nonsense, do consider the following points:

  • almost all contributions to the kernel are done by large companies, and if you are not an employee in one of them, you are likely not going to be able to do much.

    This can be inferred by the fact that the devices/ directory is by far the largest in the kernel.

    The kernel is of course just an interface to hardware, and the hardware developers start developing their kernel stuff even before specs are publicly released, both to help with hardware development and to have things working when the announcement is made.

    Furthermore, I believe that there are in-tree devices which have never been properly publicly documented. Linus is of course fine with this, since code == documentation for him, but it is not as easy for mere mortals.

    There are some less hardware bound higher level layers in the kernel which might not require being in a hardware company, and a few people must be living off it.

    But of course, those are heavily motivated by the underlying hardware characteristics, and it is very likely that most of the people working there were previously at a hardware company.

    In that sense, therefore, the kernel is not as open as one might want to believe.

    Of course, if there is some super useful and undocumented hardware that is just waiting there to be reverse engineered, then that’s a much juicier target :-)

  • it is impossible to become rich with this knowledge.

    This is partly implied by the fact that you need to be in a big company to make useful low level things, and therefore you will only be a tiny cog in the engine.

    The key problem is that the entry cost of hardware design is just too insanely high for startups in general.

  • Is learning this the most useful thing that you think can do for society?

    Or are you just learning it for job security and having a nice sounding title?

    I’m not a huge fan of the person, but I think Jobs said it right: https://www.youtube.com/watch?v=FF-tKLISfPE

    First determine the useful goal, and then backtrack down to the most efficient thing you can do to reach it.

  • there are two things that sadden me compared to physics-based engineering:

    • you will never become eternally famous. All tech disappears sooner or later, while laws of nature, at least as useful approximations, stay unchanged.

    • every problem that you face is caused by imperfections introduced by other humans.

      It is much easier to accept limitations of physics, and even natural selection in biology, which are not produced by a sentient being (?).

    Physics-based engineering, just like low level hardware, is of course completely closed source however, since wrestling against the laws of physics is about the most expensive thing humans can do, so there’s also a downside to it.

Are you fine with those points, and ready to continue wasting your life with this crap?

Good. In that case, read on, and let’s have some fun together ;-)

Related: [soft-topics].

2.2. QEMU Buildroot setup

2.2.1. QEMU Buildroot setup getting started

This setup has been tested on Ubuntu 20.04.

The Buildroot build is already broken on Ubuntu 21.04 onwards: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/155, just use the Docker host setup setup in that case. We could fix it on Ubuntu 21.04, but it will break again inevitably later on.

For other host operating systems see: [supported-hosts].

Reserve 12Gb of disk and run:

git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
./setup
./build --download-dependencies qemu-buildroot
./run

You don’t need to clone recursively even though we have .git submodules: download-dependencies fetches just the submodules that you need for this build to save time.

If something goes wrong, see: [common-build-issues] and use our issue tracker: https://github.com/cirosantilli/linux-kernel-module-cheat/issues

The initial build will take a while (30 minutes to 2 hours) to clone and build, see [benchmark-builds] for more details.

If you don’t want to wait, you could also try the following faster but much more limited methods:

but you will soon find that they are simply not enough if you anywhere near serious about systems programming.

After ./run, QEMU opens up leaving you in the /lkmc/ directory, and you can start playing with the kernel modules inside the simulated system:

insmod hello.ko
insmod hello2.ko
rmmod hello
rmmod hello2

This should print to the screen:

hello init
hello2 init
hello cleanup
hello2 cleanup

which are printk messages from init and cleanup methods of those modules.

Sources:

Quit QEMU with:

Ctrl-A X

All available modules can be found in the kernel_modules directory.

It is super easy to build for different CPU architectures, just use the --arch option:

./setup
./build --arch aarch64 --download-dependencies qemu-buildroot
./run --arch aarch64

To avoid typing --arch aarch64 many times, you can set the default arch as explained at: [default-command-line-arguments]

I now urge you to read the following sections which contain widely applicable information:

Once you use GDB step debug and tmux, your terminal will look a bit like this:

[    1.451857] input: AT Translated Set 2 keyboard as /devices/platform/i8042/s1loading @0xffffffffc0000000: ../kernel_modules-1.0//timer.ko
[    1.454310] ledtrig-cpu: registered to indicate activity on CPUs             (gdb) b lkmc_timer_callback
[    1.455621] usbcore: registered new interface driver usbhid                  Breakpoint 1 at 0xffffffffc0000000: file /home/ciro/bak/git/linux-kernel-module
[    1.455811] usbhid: USB HID core driver                                      -cheat/out/x86_64/buildroot/build/kernel_modules-1.0/./timer.c, line 28.
[    1.462044] NET: Registered protocol family 10                               (gdb) c
[    1.467911] Segment Routing with IPv6                                        Continuing.
[    1.468407] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    1.470859] NET: Registered protocol family 17                               Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 <mytimer>)
[    1.472017] 9pnet: Installing 9P2000 support                                     at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    1.475461] sched_clock: Marking stable (1473574872, 0)->(1554017593, -80442)kernel_modules-1.0/./timer.c:28
[    1.479419] ALSA device list:                                                28      {
[    1.479567]   No soundcards found.                                           (gdb) c
[    1.619187] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100                 Continuing.
[    1.622954] ata2.00: configured for MWDMA2
[    1.644048] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ P5Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 <mytimer>)
[    1.741966] tsc: Refined TSC clocksource calibration: 2904.010 MHz               at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    1.742796] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x29dc0f4skernel_modules-1.0/./timer.c:28
[    1.743648] clocksource: Switched to clocksource tsc                         28      {
[    2.072945] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8043(gdb) bt
[    2.078641] EXT4-fs (vda): couldn't mount as ext3 due to feature incompatibis#0  lkmc_timer_callback (data=0xffffffffc0002000 <mytimer>)
[    2.080350] EXT4-fs (vda): mounting ext2 file system using the ext4 subsystem    at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    2.088978] EXT4-fs (vda): mounted filesystem without journal. Opts: (null)  kernel_modules-1.0/./timer.c:28
[    2.089872] VFS: Mounted root (ext2 filesystem) readonly on device 254:0.    #1  0xffffffff810ab494 in call_timer_fn (timer=0xffffffffc0002000 <mytimer>,
[    2.097168] devtmpfs: mounted                                                    fn=0xffffffffc0000000 <lkmc_timer_callback>) at kernel/time/timer.c:1326
[    2.126472] Freeing unused kernel memory: 1264K                              #2  0xffffffff810ab71f in expire_timers (head=<optimized out>,
[    2.126706] Write protecting the kernel read-only data: 16384k                   base=<optimized out>) at kernel/time/timer.c:1363
[    2.129388] Freeing unused kernel memory: 2024K                              #3  __run_timers (base=<optimized out>) at kernel/time/timer.c:1666
[    2.139370] Freeing unused kernel memory: 1284K                              #4  run_timer_softirq (h=<optimized out>) at kernel/time/timer.c:1692
[    2.246231] EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck isd#5  0xffffffff81a000cc in __do_softirq () at kernel/softirq.c:285
[    2.259574] EXT4-fs (vda): re-mounted. Opts: block_validity,barrier,user_xatr#6  0xffffffff810577cc in invoke_softirq () at kernel/softirq.c:365
hello S98                                                                       #7  irq_exit () at kernel/softirq.c:405
                                                                                #8  0xffffffff818021ba in exiting_irq () at ./arch/x86/include/asm/apic.h:541
Apr 15 23:59:23 login[49]: root login on 'console'                              #9  smp_apic_timer_interrupt (regs=<optimized out>)
hello /root/.profile                                                                at arch/x86/kernel/apic/apic.c:1052
# insmod /timer.ko                                                              #10 0xffffffff8180190f in apic_timer_interrupt ()
[    6.791945] timer: loading out-of-tree module taints kernel.                     at arch/x86/entry/entry_64.S:857
# [    7.821621] 4294894248                                                     #11 0xffffffff82003df8 in init_thread_union ()
[    8.851385] 4294894504                                                       #12 0x0000000000000000 in ?? ()
                                                                                (gdb)

2.2.2. How to hack stuff

Besides a seamless initial build, this project also aims to make it effortless to modify and rebuild several major components of the system, to serve as an awesome development setup.

2.2.2.1. Your first Linux kernel hack

Let’s hack up the Linux kernel entry point, which is an easy place to start.

Open the file:

vim submodules/linux/init/main.c

and find the start_kernel function, then add there a:

pr_info("I'VE HACKED THE LINUX KERNEL!!!");

Then rebuild the Linux kernel, quit QEMU and reboot the modified kernel:

./build-linux
./run

and, surely enough, your message has appeared at the beginning of the boot:

<6>[    0.000000] I'VE HACKED THE LINUX KERNEL!!!

So you are now officially a Linux kernel hacker, way to go!

We could have used just build to rebuild the kernel as in the initial build instead of build-linux, but building just the required individual components is preferred during development:

  • saves a few seconds from parsing Make scripts and reading timestamps

  • makes it easier to understand what is being done in more detail

  • allows passing more specific options to customize the build

The build script is just a lightweight wrapper that calls the smaller build scripts, and you can see what ./build does with:

./build --dry-run

When you reach difficulties, QEMU makes it possible to easily GDB step debug the Linux kernel source code, see: Section 3, “GDB step debug”.

2.2.2.2. Your first kernel module hack

Edit kernel_modules/hello.c to contain:

pr_info("hello init hacked\n");

and rebuild with:

./build-modules

Now there are two ways to test it out: the fast way, and the safe way.

The fast way is, without quitting or rebooting QEMU, just directly re-insert the module with:

insmod /mnt/9p/out_rootfs_overlay/lkmc/hello.ko

and the new pr_info message should now show on the terminal at the end of the boot.

This works because we have a 9P mount there setup by default, which mounts the host directory that contains the build outputs on the guest:

ls "$(./getvar out_rootfs_overlay_dir)"

The fast method is slightly risky because your previously insmodded buggy kernel module attempt might have corrupted the kernel memory, which could affect future runs.

Such failures are however unlikely, and you should be fine if you don’t see anything weird happening.

The safe way, is to fist quit QEMU, rebuild the modules, put them in the root filesystem, and then reboot:

./build-modules
./build-buildroot
./run --eval-after 'insmod hello.ko'

./build-buildroot is required after ./build-modules because it re-generates the root filesystem with the modules that we compiled at ./build-modules.

You can see that ./build does that as well, by running:

./build --dry-run

--eval-after is optional: you could just type insmod hello.ko in the terminal, but this makes it run automatically at the end of boot, and then drops you into a shell.

If the guest and host are the same arch, typically x86_64, you can speed up boot further with KVM:

./run --kvm

All of this put together makes the safe procedure acceptably fast for regular development as well.

It is also easy to GDB step debug kernel modules with our setup, see: Section 3.4, “GDB step debug kernel module”.

2.2.2.3. Your first glibc hack

We use glibc as our default libc now, and it is tracked as an unmodified submodule at submodules/glibc, at the exact same version that Buildroot has it, which can be found at: package/glibc/glibc.mk. Buildroot 2018.05 applies no patches.

Let’s hack up the puts function:

./build-buildroot -- glibc-reconfigure

with the patch:

diff --git a/libio/ioputs.c b/libio/ioputs.c
index 706b20b492..23185948f3 100644
--- a/libio/ioputs.c
+++ b/libio/ioputs.c
@@ -38,8 +38,9 @@ _IO_puts (const char *str)
   if ((_IO_vtable_offset (_IO_stdout) != 0
        || _IO_fwide (_IO_stdout, -1) == -1)
       && _IO_sputn (_IO_stdout, str, len) == len
+      && _IO_sputn (_IO_stdout, " hacked", 7) == 7
       && _IO_putc_unlocked ('\n', _IO_stdout) != EOF)
-    result = MIN (INT_MAX, len + 1);
+    result = MIN (INT_MAX, len + 1 + 7);

   _IO_release_lock (_IO_stdout);
   return result;

And then:

./run --eval-after './c/hello.out'

outputs:

hello hacked

Lol!

We can also test our hacked glibc on User mode simulation with:

./run --userland userland/c/hello.c

I just noticed that this is actually a good way to develop glibc for other archs.

In this example, we got away without recompiling the userland program because we made a change that did not affect the glibc ABI, see this answer for an introduction to ABI stability: https://stackoverflow.com/questions/2171177/what-is-an-application-binary-interface-abi/54967743#54967743

Note that for arch agnostic features that don’t rely on bleeding kernel changes that you host doesn’t yet have, you can develop glibc natively as explained at:

Tested on a30ed0f047523ff2368d421ee2cce0800682c44e + 1.

2.2.2.4. Your first Binutils hack

Have you ever felt that a single inc instruction was not enough? Really? Me too!

So let’s hack the [gnu-gas-assembler], which is part of GNU Binutils, to add a new shiny version of inc called…​ myinc!

GCC uses GNU GAS as its backend, so we will test out new mnemonic with an [gcc-inline-assembly] test program: userland/arch/x86_64/binutils_hack.c, which is just a copy of userland/arch/x86_64/binutils_nohack.c but with myinc instead of inc.

The inline assembly is disabled with an #ifdef, so first modify the source to enable that.

Then, try to build userland:

./build-userland

and watch it fail with:

binutils_hack.c:8: Error: no such instruction: `myinc %rax'

Now, edit the file

vim submodules/binutils-gdb/opcodes/i386-tbl.h

and add a copy of the "inc" instruction just next to it, but with the new name "myinc":

diff --git a/opcodes/i386-tbl.h b/opcodes/i386-tbl.h
index af583ce578..3cc341f303 100644
--- a/opcodes/i386-tbl.h
+++ b/opcodes/i386-tbl.h
@@ -1502,6 +1502,19 @@ const insn_template i386_optab[] =
     { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
 	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
+  { "myinc", 1, 0xfe, 0x0, 1,
+    { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } },
+    { 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0 },
+    { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
+	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
   { "sub", 2, 0x28, None, 1,
     { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

Finally, rebuild Binutils, userland and test our program with User mode simulation:

./build-buildroot -- host-binutils-rebuild
./build-userland --static
./run --static --userland userland/arch/x86_64/binutils_hack.c

and we se that myinc worked since the assert did not fail!

Tested on b60784d59bee993bf0de5cde6c6380dd69420dda + 1.

2.2.2.5. Your first GCC hack

OK, now time to hack GCC.

For convenience, let’s use the User mode simulation.

If we run the program userland/c/gcc_hack.c:

./build-userland --static
./run --static --userland userland/c/gcc_hack.c

it produces the normal boring output:

i = 2
j = 0

So how about we swap ++ and -- to make things more fun?

Open the file:

vim submodules/gcc/gcc/c/c-parser.c

and find the function c_parser_postfix_expression_after_primary.

In that function, swap case CPP_PLUS_PLUS and case CPP_MINUS_MINUS:

diff --git a/gcc/c/c-parser.c b/gcc/c/c-parser.c
index 101afb8e35f..89535d1759a 100644
--- a/gcc/c/c-parser.c
+++ b/gcc/c/c-parser.c
@@ -8529,7 +8529,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 		expr.original_type = DECL_BIT_FIELD_TYPE (field);
 	    }
 	  break;
-	case CPP_PLUS_PLUS:
+	case CPP_MINUS_MINUS:
 	  /* Postincrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)->get_finish ();
@@ -8548,7 +8548,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 	  expr.original_code = ERROR_MARK;
 	  expr.original_type = NULL;
 	  break;
-	case CPP_MINUS_MINUS:
+	case CPP_PLUS_PLUS:
 	  /* Postdecrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)->get_finish ();

Now rebuild GCC, the program and re-run it:

./build-buildroot -- host-gcc-final-rebuild
./build-userland --static
./run --static --userland userland/c/gcc_hack.c

and the new ouptut is now:

i = 2
j = 0

We need to use the ugly -final thing because GCC has to packages in Buildroot, -initial and -final: https://stackoverflow.com/questions/54992977/how-to-select-an-override-srcdir-source-for-gcc-when-building-buildroot No one is able to example precisely with a minimal example why this is required:

2.2.3. About the QEMU Buildroot setup

What QEMU and Buildroot are:

This is our reference setup, and the best supported one, use it unless you have good reason not to.

It was historically the first one we did, and all sections have been tested with this setup unless explicitly noted.

Read the following sections for further introductory material:

2.3. Dry run to get commands for your project

One of the major features of this repository is that we try to support the --dry-run option really well for all scripts.

This option, as the name suggests, outputs the external commands that would be run (or more precisely: equivalent commands), without actually running them.

This allows you to just clone this repository and get full working commands to integrate into your project, without having to build or use this setup further!

For example, we can obtain a QEMU run for the file userland/c/hello.c in User mode simulation by adding --dry-run to the normal command:

./run --dry-run --userland userland/c/hello.c

which as of LKMC a18f28e263c91362519ef550150b5c9d75fa3679 + 1 outputs:

+ /path/to/linux-kernel-module-cheat/out/qemu/default/opt/x86_64-linux-user/qemu-x86_64 \
  -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \
  -r 5.2.1 \
  -seed 0 \
  -trace enable=load_file,file=/path/to/linux-kernel-module-cheat/out/run/qemu/x86_64/0/trace.bin \
  -cpu max \
  /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/hello.out \
;

So observe that the command contains:

  • +: sign to differentiate it from program stdout, much like bash -x output. This is not a valid part of the generated Bash command however.

  • the actual command nicely, indented and with arguments broken one per line, but with continuing backslashes so you can just copy paste into a terminal

    For setups that don’t support the newline e.g. Eclipse debugging, you can turn them off with --print-cmd-oneline

  • ;: both a valid part of the Bash command, and a visual mark the end of the command

For the specific case of running emulators such as QEMU, the last command is also automatically placed in a file for your convenience and later inspection:

cat "$(./getvar run_dir)/run.sh"

Since we need this so often, the last run command is also stored for convenience at:

cat out/run.sh

although this won’t of course work well for [simultaneous-runs].

Furthermore, --dry-run also automatically specifies, in valid Bash shell syntax:

  • environment variables used to run the command with syntax + ENV_VAR_1=abc ENV_VAR_2=def ./some/command

  • change in working directory with + cd /some/new/path && ./some/command

2.4. gem5 Buildroot setup

2.4.1. About the gem5 Buildroot setup

This setup is like the QEMU Buildroot setup, but it uses gem5 instead of QEMU as a system simulator.

QEMU tries to run as fast as possible and give correct results at the end, but it does not tell us how many CPU cycles it takes to do something, just the number of instructions it ran. This kind of simulation is known as functional simulation.

The number of instructions executed is a very poor estimator of performance because in modern computers, a lot of time is spent waiting for memory requests rather than the instructions themselves.

gem5 on the other hand, can simulate the system in more detail than QEMU, including:

  • simplified CPU pipeline

  • caches

  • DRAM timing

and can therefore be used to estimate system performance, see: Section 24.2, “gem5 run benchmark” for an example.

The downside of gem5 much slower than QEMU because of the greater simulation detail.

See gem5 vs QEMU for a more thorough comparison.

2.4.2. gem5 Buildroot setup getting started

For the most part, if you just add the --emulator gem5 option or *-gem5 suffix to all commands and everything should magically work.

If you haven’t built Buildroot yet for QEMU Buildroot setup, you can build from the beginning with:

./setup
./build --download-dependencies gem5-buildroot
./run --emulator gem5

If you have already built previously, don’t be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so ./build will be fast.

Remember that the gem5 boot is considerably slower than QEMU since the simulation is more detailed.

If you have a relatively new GCC version and the gem5 build fails on your machine, see: [gem5-build-broken-on-recent-compiler-version].

To get a terminal, either open a new shell and run:

./gem5-shell

You can quit the shell without killing gem5 by typing tilde followed by a period:

~.

If you are inside tmux, which I highly recommend, you can both run gem5 stdout and open the guest terminal on a split window with:

./run --emulator gem5 --tmux

At the end of boot, it might not be very clear that you have the shell since some printk messages may appear in front of the prompt like this:

# <6>[    1.215329] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1cd486fa865, max_idle_ns: 440795259574 ns
<6>[    1.215351] clocksource: Switched to clocksource tsc

but if you look closely, the PS1 prompt marker # is there already, just hit enter and a clear prompt line will appear.

If you forgot to open the shell and gem5 exit, you can inspect the terminal output post-mortem at:

less "$(./getvar --emulator gem5 m5out_dir)/system.pc.com_1.device"

More gem5 information is present at: Section 24, “gem5”

Good next steps are:

  • gem5 run benchmark: how to run a benchmark in gem5 full system, including how to boot Linux, checkpoint and restore to skip the boot on a fast CPU

  • m5out directory: understand the output files that gem5 produces, which contain information about your run

  • m5ops: magic guest instructions used to control gem5

  • [add-new-files-to-the-buildroot-image]: how to add your own files to the image if you have a benchmark that we don’t already support out of the box (also send a pull request!)

2.5. Docker host setup

This repository has been tested inside clean Docker containers.

This is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: [supported-hosts].

For example, to do a QEMU Buildroot setup inside Docker, run:

sudo apt-get install docker
./setup
./run-docker create && \
./run-docker sh -- ./build --download-dependencies qemu-buildroot
./run-docker

You are now left inside a shell in the Docker! From there, just run as usual:

./run

The host git top level directory is mounted inside the guest with a Docker volume, which means for example that you can use your host’s GUI text editor directly on the files. Just don’t forget that if you nuke that directory on the guest, then it gets nuked on the host as well!

Command breakdown:

  • ./run-docker create: create the image and container.

    Needed only the very first time you use Docker, or if you run ./run-docker DESTROY to restart for scratch, or save some disk space.

    The image and container name is lkmc. The container shows under:

    docker ps -a

    and the image shows under:

    docker images
  • ./run-docker: open a shell on the container.

    If it has not been started previously, start it. This can also be done explicitly with:

    ./run-docker start

    Quit the shell as usual with Ctrl-D

    This can be called multiple times from different host terminals to open multiple shells.

  • ./run-docker stop: stop the container.

    This might save a bit of CPU and RAM once you stop working on this project, but it should not be a lot.

  • ./run-docker DESTROY: delete the container and image.

    This doesn’t really clean the build, since we mount the guest’s working directory on the host git top-level, so you basically just got rid of the apt-get installs.

    To actually delete the Docker build, run on host:

    # sudo rm -rf out.docker

To use GDB step debug from inside Docker, you need a second shell inside the container. You can either do that from another shell with:

./run-docker

or even better, by starting a tmux session inside the container. We install tmux by default in the container.

You can also start a second shell and run a command in it at the same time with:

./run-docker sh -- ./run-gdb start_kernel

To use QEMU graphic mode from Docker, run:

./run --graphic --vnc

and then on host:

sudo apt-get install vinagre
./vnc

TODO make files created inside Docker be owned by the current user in host instead of root:

2.6. Prebuilt setup

2.6.1. About the prebuilt setup

This setup uses prebuilt binaries that we upload to GitHub from time to time.

We don’t currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.

Our prebuilts currently include:

For more details, see our our release procedure.

Advantage of this setup: saves time and disk space on the initial install, which is expensive in largely due to building the toolchain.

The limitations are severe however:

  • can’t GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: [prebuilt-toolchain].

    Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.

  • you won’t get the latest version of this repository. Our [travis] attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us anyway.

  • gem5 is not currently supported. The major blocking point is how to avoid distributing the kernel images twice: once for gem5 which uses vmlinux, and once for QEMU which uses arch/* images, see also:

This setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the full featured setup while you take a coffee or a nap? :-)

2.6.2. Prebuilt setup getting started

Checkout to the latest tag and use the Ubuntu packaged QEMU to boot Linux:

sudo apt-get install qemu-system-x86
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
git checkout "$(git rev-list --tags --max-count=1)"
./release-download-latest
unzip lkmc-*.zip
./run --qemu-which host

You have to checkout to the latest tag to ensure that the scripts match the release format: https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git

This is known not to work for aarch64 on an Ubuntu 16.04 host with QEMU 2.5.0, presumably because QEMU is too old, the terminal does not show any output. I haven’t investigated why.

Or to run a baremetal example instead:

./run \
  --arch aarch64 \
  --baremetal userland/c/hello.c \
  --qemu-which host \
;

Be saner and use our custom built QEMU instead:

./setup
./build --download-dependencies qemu
./run

To build the kernel modules as in Your first kernel module hack do:

git submodule update --depth 1 --init --recursive "$(./getvar linux_source_dir)"
./build-linux --no-modules-install -- modules_prepare
./build-modules --gcc-which host
./run

TODO: for now the only way to test those modules out without building Buildroot is with 9p, since we currently rely on Buildroot to manipulate the root filesystem.

Command explanation:

  • modules_prepare does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.

  • --gcc-which host selects your host Ubuntu packaged GCC, since you don’t have the Buildroot toolchain

  • --no-modules-install is required otherwise the make modules_install target we run by default fails, since the kernel wasn’t built

To modify the Linux kernel, build and use it as usual:

git submodule update --depth 1 --init --recursive "$(./getvar linux_source_dir)"
./build-linux
./run

2.7. Host kernel module setup

THIS IS DANGEROUS (AND FUN), YOU HAVE BEEN WARNED

This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.

It has however severe limitations:

  • can’t control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since the Linux kernel does not have a stable kernel module API.

  • bugs can easily break you system. E.g.:

    • segfaults can trivially lead to a kernel crash, and require a reboot

    • your disk could get erased. Yes, this can also happen with sudo from userland. But you should not use sudo when developing newbie programs. And for the kernel you don’t have the choice not to use sudo.

    • even more subtle system corruption such as not being able to rmmod

  • can’t control which hardware is used, notably the CPU architecture

  • can’t step debug it with GDB easily. The alternatives are JTAG or KGDB, but those are less reliable, and require extra hardware.

Still interested?

./build-modules --host

Compilation will likely fail for some modules because of kernel or toolchain differences that we can’t control on the host.

The best workaround is to compile just your modules with:

./build-modules --host -- hello hello2

which is equivalent to:

./build-modules \
  --gcc-which host \
  --host \
  -- \
  kernel_modules/hello.c \
  kernel_modules/hello2.c \
;

Or just remove the .c extension from the failing files and try again:

cd "$(./getvar kernel_modules_source_dir)"
mv broken.c broken.c~

Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:

cd "$(./getvar kernel_modules_build_host_subdir)"
sudo insmod hello.ko

# Our module is there.
sudo lsmod | grep hello

# Last message should be: hello init
dmesg -T

sudo rmmod hello

# Last message should be: hello exit
dmesg -T

# Not present anymore
sudo lsmod | grep hello

2.7.1. Hello host

Minimal host build system example:

cd hello_host_kernel_module
make
sudo insmod hello.ko
dmesg
sudo rmmod hello.ko
dmesg

2.8. Userland setup

2.8.1. About the userland setup

In order to test the kernel and emulators, userland content in the form of executables and scripts is of course required, and we store it mostly under:

When we started this repository, it only contained content that interacted very closely with the kernel, or that had required performance analysis.

However, we soon started to notice that this had an increasing overlap with other userland test repositories: we were duplicating build and test infrastructure and even some examples.

Therefore, we decided to consolidate other userland tutorials that we had scattered around into this repository.

Notable userland content included / moving into this repository includes:

2.8.2. Userland setup getting started

There are several ways to run our [userland-content], notably:

2.8.2.1. Userland setup getting started natively

With this setup, we will use the host toolchain and execute executables directly on the host.

No toolchain build is required, so you can just download your distro toolchain and jump straight into it.

Build, run and example, and clean it in-tree with:

sudo apt-get install gcc
cd userland
./build c/hello
./c/hello.out
./build --clean

Build an entire directory and test it:

cd userland
./build c
./test c

Build the current directory and test it:

cd userland/c
./build
./test

As mentioned at [userland-libs-directory], tests under userland/libs require certain optional libraries to be installed, and are not built or tested by default.

You can install those libraries with:

cd linux-kernel-module-cheat
./setup
./build --download-dependencies userland-host

and then build the examples and test with:

./build --package-all
./test --package-all

Pass custom compiler options:

./build --ccflags='-foptimize-sibling-calls -foptimize-strlen' --force-rebuild

Here we used --force-rebuild to force rebuild since the sources weren’t modified since the last build.

Some CLI options have more specialized flags, e.g. -O for the [optimization-level-of-a-build]:

./build --optimization-level 3 --force-rebuild

See also User mode static executables for --static.

The build scripts inside userland/ are just symlinks to build-userland-in-tree which you can also use from toplevel as:

./build-userland-in-tree
./build-userland-in-tree userland/c
./build-userland-in-tree userland/c/hello.c

build-userland-in-tree is in turn just a thin wrapper around build-userland:

./build-userland --gcc-which host --in-tree userland/c

So you can use any option supported by build-userland script freely with build-userland-in-tree and build.

The situation is analogous for userland/test, test-executables-in-tree and test-executables, which are further documented at: Section 11.2, “User mode tests”.

Do a more clean out-of-tree build instead and run the program:

./build-userland --gcc-which host --userland-build-id host
./run --emulator native --userland userland/c/hello.c --userland-build-id host

Here we:

  • put the host executables in a separate build variant to avoid conflict with Buildroot builds.

  • ran with the --emulator native option to run the program natively

In this case you can debub the program with:

./run --debug-vm --emulator native --userland userland/c/hello.c --userland-build-id host

as shown at: Section 23.8, “Debug the emulator”, although direct GDB host usage works as well of course.

2.8.2.2. Userland setup getting started with prebuilt toolchain and QEMU user mode

If you are lazy to built the Buildroot toolchain and QEMU, but want to run e.g. ARM [userland-assembly] in User mode simulation, you can get away on Ubuntu 18.04 with just:

sudo apt-get install gcc-aarch64-linux-gnu qemu-system-aarch64
./build-userland \
  --arch aarch64 \
  --gcc-which host \
  --userland-build-id host \
;
./run \
  --arch aarch64 \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

where:

This present the usual trade-offs of using prebuilts as mentioned at: Section 2.6, “Prebuilt setup”.

Other functionality are analogous, e.g. testing:

./test-executables \
  --arch aarch64 \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
;
./run \
  --arch aarch64 \
  --gdb \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;
2.8.2.3. Userland setup getting started full system

First ensure that QEMU Buildroot setup is working.

After doing that setup, you can already execute your userland programs from inside QEMU: the only missing step is how to rebuild executables and run them.

And the answer is exactly analogous to what is shown at: Section 2.2.2.2, “Your first kernel module hack”

For example, if we modify userland/c/hello.c to print out something different, we can just rebuild it with:

./build-userland

Source: build-userland. ./build calls that script automatically for us when doing the initial full build.

Now, run the program either without rebooting use the 9P mount:

/mnt/9p/out_rootfs_overlay/c/hello.out

or shutdown QEMU, add the executable to the root filesystem:

./build-buildroot

reboot and use the root filesystem as usual:

./hello.out

2.9. Baremetal setup

2.9.1. About the baremetal setup

This setup does not use the Linux kernel nor Buildroot at all: it just runs your very own minimal OS.

x86_64 is not currently supported, only arm and aarch64: I had made some x86 bare metal examples at: cirosantilli/x86-bare-metal-examples but I’m lazy to port them here now. Pull requests are welcome.

The main reason this setup is included in this project, despite the word "Linux" being on the project name, is that a lot of the emulator boilerplate can be reused for both use cases.

This setup allows you to make a tiny OS and that runs just a few instructions, use it to fully control the CPU to better understand the simulators for example, or develop your own OS if you are into that.

You can also use C and a subset of the C standard library because we enable Newlib by default. See also:

Our C bare-metal compiler is built with crosstool-NG. If you have already built Buildroot previously, you will end up with two GCCs installed. Unfortunately I don’t see a solution for this, since we need separate toolchains for Newlib on baremetal and glibc on Linux: https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869

2.9.2. Baremetal setup getting started

Every .c file inside baremetal/ and .S file inside baremetal/arch/<arch>/ generates a separate baremetal image.

For example, to run baremetal/arch/aarch64/dump_regs.c in QEMU do:

./setup
./build --arch aarch64 --download-dependencies qemu-baremetal
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c

And the terminal prints the values of certain system registers. This example prints registers that are only accessible from EL1 or higher, and thus could not be run in userland.

In addition to the examples under baremetal/, several of the userland examples can also be run in baremetal! This is largely due to the awesomeness of Newlib.

The examples that work include most C examples that don’t rely on complicated syscalls such as threads, and almost all the [userland-assembly] examples.

The exact list of userland programs that work in baremetal is specified in [path-properties] with the baremetal property, but you can also easily find it out with a baremetal test dry run:

./test-executables --arch aarch64 --dry-run --mode baremetal

For example, we can run the C hello world userland/c/hello.c simply as:

./run --arch aarch64 --baremetal userland/c/hello.c

and that outputs to the serial port the string:

hello

which QEMU shows on the host terminal.

To modify a baremetal program, simply edit the file, e.g.

vim userland/c/hello.c

and rebuild:

./build-baremetal --arch aarch64
./run --arch aarch64 --baremetal userland/c/hello.c

./build qemu-baremetal that we run previously is only needed for the initial build. That script calls build-baremetal for us, in addition to building prerequisites such as QEMU and crosstool-NG.

./build-baremetal uses crosstool-NG, and so it must be preceded by build-crosstool-ng, which ./build qemu-baremetal also calls.

Now let’s run userland/arch/aarch64/add.S:

./run --arch aarch64 --baremetal userland/arch/aarch64/add.S

This time, the terminal does not print anything, which indicates success: if you look into the source, you will see that we just have an assertion there.

You can see a sample assertion fail in userland/c/assert_fail.c:

./run --arch aarch64 --baremetal userland/c/assert_fail.c

and the terminal contains:

lkmc_exit_status_134
error: simulation error detected by parsing logs

and the exit status of our script is 1:

echo $?

You can run all the baremetal examples in one go and check that all assertions passed with:

./test-executables --arch aarch64 --mode baremetal

To use gem5 instead of QEMU do:

./setup
./build --download-dependencies gem5-baremetal
./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5

and then as usual open a shell with:

./gem5-shell

Or as usual, tmux users can do both in one go with:

./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --tmux

TODO: the carriage returns are a bit different than in QEMU, see: [gem5-baremetal-carriage-return].

Note that ./build-baremetal requires the --emulator gem5 option, and generates separate executable images for both, as can be seen from:

echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator qemu image)"
echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 image)"

This is unlike the Linux kernel that has a single image for both QEMU and gem5:

echo "$(./getvar --arch aarch64 --emulator qemu image)"
echo "$(./getvar --arch aarch64 --emulator gem5 image)"

The reason for that is that on baremetal we don’t parse the device tress from memory like the Linux kernel does, which tells the kernel for example the UART address, and many other system parameters.

gem5 also supports the RealViewPBX machine, which represents an older hardware compared to the default VExpress_GEM5_V1:

./build-baremetal --arch aarch64 --emulator gem5 --machine RealViewPBX
./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX

This generates yet new separate images with new magic constants:

echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine VExpress_GEM5_V1 image)"
echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX      image)"

But just stick to newer and better VExpress_GEM5_V1 unless you have a good reason to use RealViewPBX.

When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: [userland-assembly].

For more information on baremetal, see the section: [baremetal].

The following subjects are particularly important:

2.10. Build the documentation

You don’t need to depend on GitHub.

For a quick and dirty build, install Asciidoctor however you like and build:

asciidoctor README.adoc
xdg-open README.html

For development, you will want to do a more controlled build with extra error checking as follows.

For the initial build do:

./setup
./build --download-dependencies docs

which also downloads build dependencies.

Then the following times just to the faster:

./build-doc

Source: build-doc

The HTML output is located at:

xdg-open out/README.html

More information about our documentation internals can be found at: [documentation]

3. GDB step debug

3.1. GDB step debug kernel boot

--gdb-wait makes QEMU and gem5 wait for a GDB connection, otherwise we could accidentally go past the point we want to break at:

./run --gdb-wait

Say you want to break at start_kernel. So on another shell:

./run-gdb start_kernel

or at a given line:

./run-gdb init/main.c:1088

Now QEMU will stop there, and you can use the normal GDB commands:

list
next
continue

See also:

3.1.1. GDB step debug kernel boot other archs

Just don’t forget to pass --arch to ./run-gdb, e.g.:

./run --arch aarch64 --gdb-wait

and:

./run-gdb --arch aarch64 start_kernel

3.1.2. Disable kernel compiler optimizations

O=0 is an impossible dream, O=2 being the default.

So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.

The -O level of some other userland content can be controlled as explained at: [optimization-level-of-a-build].

3.2. GDB step debug kernel post-boot

Let’s observe the kernel write system call as it reacts to some userland actions.

Start QEMU with just:

./run

and after boot inside a shell run:

./count.sh

which counts to infinity to stdout. Source: rootfs_overlay/lkmc/count.sh.

Then in another shell, run:

./run-gdb

and then hit:

Ctrl-C
break __x64_sys_write
continue
continue
continue

And you now control the counting on the first shell from GDB!

Before v4.17, the symbol name was just sys_write, the change happened at d5a00528b58cdb2c71206e18bd021e34c4eab878. As of Linux v 4.19, the function is called sys_write in arm, and __arm64_sys_write in aarch64. One good way to find it if the name changes again is to try:

rbreak .*sys_write

or just have a quick look at the sources!

When you hit Ctrl-C, if we happen to be inside kernel code at that point, which is very likely if there are no heavy background tasks waiting, and we are just waiting on a sleep type system call of the command prompt, we can already see the source for the random place inside the kernel where we stopped.

3.3. tmux

tmux just makes things even more fun by allowing us to see both the terminal for:

at once without dragging windows around!

First start tmux with:

tmux

Now that you are inside a shell inside tmux, you can start GDB simply with:

./run --gdb

which is just a convenient shortcut for:

./run --gdb-wait --tmux --tmux-args start_kernel

This splits the terminal into two panes:

  • left: usual QEMU with terminal

  • right: GDB

and focuses on the GDB pane.

Now you can navigate with the usual tmux shortcuts:

  • switch between the two panes with: Ctrl-B O

  • close either pane by killing its terminal with Ctrl-D as usual

See the tmux manual for further details:

man tmux

To start again, switch back to the QEMU pane with Ctrl-O, kill the emulator, and re-run:

./run --gdb

This automatically clears the GDB pane, and starts a new one.

The option --tmux-args determines which options will be passed to the program running on the second tmux pane, and is equivalent to:

This is equivalent to:

./run --gdb-wait
./run-gdb start_kernel

Due to Python’s CLI parsing quicks, if the run-gdb arguments start with a dash -, you have to use the = sign, e.g. to GDB step debug early boot:

./run --gdb --tmux-args=--no-continue

3.3.1. tmux gem5

If you are using gem5 instead of QEMU, --tmux has a different effect by default: it opens the gem5 terminal instead of the debugger:

./run --emulator gem5 --tmux

To open a new pane with GDB instead of the terminal, use:

./run --gdb

which is equivalent to:

./run --emulator gem5 --gdb-wait --tmux --tmux-args start_kernel --tmux-program gdb

--tmux-program implies --tmux, so we can just write:

./run --emulator gem5 --gdb-wait --tmux-program gdb

If you also want to see both GDB and the terminal with gem5, then you will need to open a separate shell manually as usual with ./gem5-shell.

From inside tmux, you can create new terminals on a new window with Ctrl-B C split a pane yet again vertically with Ctrl-B % or horizontally with Ctrl-B ".

3.4. GDB step debug kernel module

Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.

So we cannot set the breakpoints before insmod.

However, the Linux kernel GDB scripts offer the lx-symbols command, which takes care of that beautifully for us.

Shell 1:

./run

Wait for the boot to end and run:

insmod timer.ko

This prints a message to dmesg every second.

Shell 2:

./run-gdb

In GDB, hit Ctrl-C, and note how it says:

scanning for modules in /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules
loading @0xffffffffc0000000: /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/timer.ko

That’s lx-symbols working! Now simply:

break lkmc_timer_callback
continue
continue
continue

and we now control the callback from GDB!

Just don’t forget to remove your breakpoints after rmmod, or they will point to stale memory locations.

TODO: why does break work_func for insmod kthread.ko not very well? Sometimes it breaks but not others.

3.4.1. GDB step debug kernel module insmodded by init on ARM

TODO on arm 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and lx-symbols fails with the message:

loading vmlinux
Traceback (most recent call last):
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 163, in invoke
    self.load_all_symbols()
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 150, in load_all_symbols
    [self.load_module_symbols(module) for module in module_list]
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 110, in load_module_symbols
    module_name = module['name'].string()
gdb.MemoryError: Cannot access memory at address 0xbf0000cc
Error occurred in Python command: Cannot access memory at address 0xbf0000cc

Can’t reproduce on x86_64 and aarch64 are fine.

It is kind of random: if you just insmod manually and then immediately ./run-gdb --arch arm, then it usually works.

But this fails most of the time: shell 1:

./run --arch arm --eval-after 'insmod hello.ko'

shell 2:

./run-gdb --arch arm

then hit Ctrl-C on shell 2, and voila.

Then:

cat /proc/modules

says that the load address is:

0xbf000000

so it is close to the failing 0xbf0000cc.

readelf:

./run-toolchain readelf -- -s "$(./getvar kernel_modules_build_subdir)/hello.ko"

does not give any interesting hits at cc, no symbol was placed that far.

3.4.2. GDB module_init

TODO find a more convenient method. We have working methods, but they are not ideal.

This is not very easy, since by the time the module finishes loading, and lx-symbols can work properly, module_init has already finished running!

Possibly asked at:

3.4.2.1. GDB module_init step into it

This is the best method we’ve found so far.

The kernel calls module_init synchronously, therefore it is not hard to step into that call.

As of 4.16, the call happens in do_one_initcall, so we can do in shell 1:

./run

shell 2 after boot finishes (because there are other calls to do_init_module at boot, presumably for the built-in modules):

./run-gdb do_one_initcall

then step until the line:

833         ret = fn();

which does the actual call, and then step into it.

For the next time, you can also put a breakpoint there directly:

./run-gdb init/main.c:833

How we found this out: first we got GDB module_init calculate entry address working, and then we did a bt. AKA cheating :-)

3.4.2.2. GDB module_init calculate entry address

This works, but is a bit annoying.

The key observation is that the load address of kernel modules is deterministic: there is a pre allocated memory region https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt "module mapping space" filled from bottom up.

So once we find the address the first time, we can just reuse it afterwards, as long as we don’t modify the module.

Do a fresh boot and get the module:

./run --eval-after './pr_debug.sh;insmod fops.ko;./linux/poweroff.out'

The boot must be fresh, because the load address changes every time we insert, even after removing previous modules.

The base address shows on terminal:

0xffffffffc0000000 .text

Now let’s find the offset of myinit:

./run-toolchain readelf -- \
  -s "$(./getvar kernel_modules_build_subdir)/fops.ko" | \
  grep myinit

which gives:

    30: 0000000000000240    43 FUNC    LOCAL  DEFAULT    2 myinit

so the offset address is 0x240 and we deduce that the function will be placed at:

0xffffffffc0000000 + 0x240 = 0xffffffffc0000240

Now we can just do a fresh boot on shell 1:

./run --eval 'insmod fops.ko;./linux/poweroff.out' --gdb-wait

and on shell 2:

./run-gdb '*0xffffffffc0000240'

GDB then breaks, and lx-symbols works.

3.4.2.3. GDB module_init break at the end of sys_init_module

TODO not working. This could be potentially very convenient.

The idea here is to break at a point late enough inside sys_init_module, at which point lx-symbols can be called and do its magic.

Beware that there are both sys_init_module and sys_finit_module syscalls, and insmod uses fmodule_init by default.

Both call do_module_init however, which is what lx-symbols hooks to.

If we try:

b sys_finit_module

then hitting:

n

does not break, and insertion happens, likely because of optimizations? Disable kernel compiler optimizations

Then we try:

b do_init_module

A naive:

fin

also fails to break!

Finally, in despair we notice that pr_debug prints the kernel load address as explained at Bypass lx-symbols.

So, if we set a breakpoint just after that message is printed by searching where that happens on the Linux source code, we must be able to get the correct load address before init_module happens.

3.4.2.4. GDB module_init add trap instruction

This is another possibility: we could modify the module source by adding a trap instruction of some kind.

This appears to be described at: https://www.linuxjournal.com/article/4525

But it refers to a gdbstart script which is not in the tree anymore and beyond my git log capabilities.

And just adding:

asm( " int $3");

directly gives an oops as I’d expect.

3.4.3. Bypass lx-symbols

Useless, but a good way to show how hardcore you are. Disable lx-symbols with:

./run-gdb --no-lxsymbols

From inside guest:

insmod timer.ko
cat /proc/modules

as mentioned at:

This will give a line of form:

fops 2327 0 - Live 0xfffffffa00000000

And then tell GDB where the module was loaded with:

Ctrl-C
add-symbol-file ../../../rootfs_overlay/x86_64/timer.ko 0xffffffffc0000000
0xffffffffc0000000

Alternatively, if the module panics before you can read /proc/modules, there is a pr_debug which shows the load address:

echo 8 > /proc/sys/kernel/printk
echo 'file kernel/module.c +p' > /sys/kernel/debug/dynamic_debug/control
./linux/myinsmod.out hello.ko

And then search for a line of type:

[   84.877482]  0xfffffffa00000000 .text

Tested on 4f4749148273c282e80b58c59db1b47049e190bf + 1.

3.5. GDB step debug early boot

TODO successfully debug the very first instruction that the Linux kernel runs, before start_kernel!

Break at the very first instruction executed by QEMU:

./run-gdb --no-continue

Note however that early boot parts appear to be relocated in memory somehow, and therefore:

  • you won’t see the source location in GDB, only assembly

  • you won’t be able to break by symbol in those early locations

Further discussion at: Linux kernel entry point.

In the specific case of gem5 aarch64 at least:

  • gem5 relocates the kernel in memory to a fixed location, see e.g. https://gem5.atlassian.net/browse/GEM5-787

  • --param 'system.workload.early_kernel_symbols=True should in theory duplicate the symbols to the correct physical location, but it was broken at one point: https://gem5.atlassian.net/browse/GEM5-785

  • gem5 executes directly from vmlinux, so there is no decompression code involved, so you actually immediately start running the "true" first instruction from head.S as described at: https://stackoverflow.com/questions/18266063/does-linux-kernel-have-main-function/33422401#33422401

  • once the MMU gets turned on at kernel symbol __primary_switched, the virtual address matches the ELF symbols, and you start seeing correct symbols without the need for early_kernel_symbols. This can be observed clearly with function_trace = True: https://stackoverflow.com/questions/64049487/how-to-trace-executed-guest-function-symbol-names-with-their-timestamp-in-gem5/64049488#64049488 which produces:

    0: _kernel_flags_le_lo32 (12500)
    12500: __crc_tcp_add_backlog (1000)
    13500: __crc_crypto_alg_tested (6500)
    20000: __crc_tcp_add_backlog (10000)
    30000: __crc_crypto_alg_tested (500)
    30500: __crc_scsi_is_host_device (5000)
    35500: __crc_crypto_alg_tested (1500)
    37000: __crc_scsi_is_host_device (4000)
    41000: __crc_crypto_alg_tested (3000)
    44000: __crc_tcp_add_backlog (263500)
    307500: __crc_crypto_alg_tested (975500)
    1283000: __crc_tcp_add_backlog (77191500)
    78474500: __crc_crypto_alg_tested (1000)
    78475500: __crc_scsi_is_host_device (19500)
    78495000: __crc_crypto_alg_tested (500)
    78495500: __crc_scsi_is_host_device (13500)
    78509000: __primary_switched (14000)
    78523000: memset (21118000)
    99641000: __primary_switched (2500)
    99643500: start_kernel (11000)

    so we see that primary_switched is the first non-trash symbol (non-crc_* and non-kernel_flags*, which are just informative symbols, not actual executable code)

3.5.1. Linux kernel entry point

As mentioned at: GDB step debug early boot, the very first kernel instructions executed appear to be placed into memory at a different location than that of the kernel ELF section.

As a result, we are unable to break on early symbols such as:

./run-gdb extract_kernel
./run-gdb main

gem5 ExecAll trace format>> however does show the right symbols however! This could be because gem5 uses vmlinux to boot, which QEMU uses the compressed version, and as mentioned on the Stack Overflow answer, the entry point is actually a tiny decompresser routine.

I also tried to hack run-gdb with:

@@ -81,7 +81,7 @@ else
 ${gdb} \
 -q \\
 -ex 'add-auto-load-safe-path $(pwd)' \\
--ex 'file vmlinux' \\
+-ex 'file arch/arm/boot/compressed/vmlinux' \\
 -ex 'target remote localhost:${port}' \\
 ${brk} \
 -ex 'continue' \\

and no I do have the symbols from arch/arm/boot/compressed/vmlinux', but the breaks still don’t work.

v4.19 also added a CONFIG_HAVE_KERNEL_UNCOMPRESSED=y option for having the kernel uncompressed which could make following the startup easier, but it is only available on s390. aarch64 however is already uncompressed by default, so might be the easiest one. See also: Section 17.20.1, “vmlinux vs bzImage vs zImage vs Image”.

You then need the associated KERNEL_UNCOMPRESSED to enable it if available:

config KERNEL_UNCOMPRESSED
    bool "None"
    depends on HAVE_KERNEL_UNCOMPRESSED
3.5.1.1. arm64 secondary CPU entry point

In gem5 aarch64 Linux v4.18, experimentally the entry point of secondary CPUs seems to be secondary_holding_pen as shown at https://gist.github.com/cirosantilli2/34a7bc450fcb6c1c1a910369be1fdd90

What happens is that:

  • the bootloader goes in in WFE

  • the kernel writes the entry point to the secondary CPU (the address of secondary_holding_pen) with CPU0 at the address given to the kernel in the cpu-release-addr of the DTB

  • the kernel wakes up the bootloader with a SEV, and the bootloader boots to the address the kernel told it

Here’s the code that writes the address and does SEV:

static int smp_spin_table_cpu_prepare(unsigned int cpu)
{
	__le64 __iomem *release_addr;

	if (!cpu_release_addr[cpu])
		return -ENODEV;

	/*
	 * The cpu-release-addr may or may not be inside the linear mapping.
	 * As ioremap_cache will either give us a new mapping or reuse the
	 * existing linear mapping, we can use it to cover both cases. In
	 * either case the memory will be MT_NORMAL.
	 */
	release_addr = ioremap_cache(cpu_release_addr[cpu],
				     sizeof(*release_addr));
	if (!release_addr)
		return -ENOMEM;

	/*
	 * We write the release address as LE regardless of the native
	 * endianess of the kernel. Therefore, any boot-loaders that
	 * read this address need to convert this address to the
	 * boot-loader's endianess before jumping. This is mandated by
	 * the boot protocol.
	 */
	writeq_relaxed(__pa_symbol(secondary_holding_pen), release_addr);
	__flush_dcache_area((__force void *)release_addr,
			    sizeof(*release_addr));

	/*
	 * Send an event to wake up the secondary CPU.
	 */
	sev();

and here’s the code that reads the value from the DTB:

static int smp_spin_table_cpu_init(unsigned int cpu)
{
	struct device_node *dn;
	int ret;

	dn = of_get_cpu_node(cpu, NULL);
	if (!dn)
		return -ENODEV;

	/*
	 * Determine the address from which the CPU is polling.
	 */
	ret = of_property_read_u64(dn, "cpu-release-addr",
				   &cpu_release_addr[cpu]);

3.5.2. Linux kernel arch-agnostic entry point

start_kernel is the first C function to be executed basically: https://stackoverflow.com/questions/18266063/does-kernel-have-main-function/33422401#33422401

For the earlier arch-specific entry point, see: Linux kernel entry point.

3.5.3. Linux kernel early boot messages

When booting Linux on a slow emulator like gem5, what you observe is that:

  • first nothing shows for a while

  • then at once, a bunch of message lines show at once followed on aarch64 Linux 5.4.3 by:

    [    0.081311] printk: console [ttyAMA0] enabled

This means of course that all the previous messages had been generated earlier and stored, but were only printed to the terminal once the terminal itself was enabled.

Notably for example the very first message:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd070]

happens very early in the boot process.

If you get a failure before that, it will be hard to see the print messages.

One possible solution is to parse the dmesg buffer, gem5 actually implements that: gem5 m5out/system.workload.dmesg file.

3.6. GDB step debug userland processes

QEMU’s -gdb GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.

You will generally want to use gdbserver for this as it is more reliable, but this method can overcome the following limitations of gdbserver:

  • the emulator does not support host to guest networking. This seems to be the case for gem5 as explained at: Section 15.3.1.3, “gem5 host to guest networking”

  • cannot see the start of the init process easily

  • gdbserver alters the working of the kernel, and makes your run less representative

Known limitations of direct userland debugging:

  • the kernel might switch context to another process or to the kernel itself e.g. on a system call, and then TODO confirm the PIC would go to weird places and source code would be missing.

    Solutions to this are being researched at: Section 3.10.1, “lx-ps”.

  • TODO step into shared libraries. If I attempt to load them explicitly:

    (gdb) sharedlibrary ../../staging/lib/libc.so.0
    No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.

    since GDB does not know that libc is loaded.

3.6.1. GDB step debug userland custom init

This is the userland debug setup most likely to work, since at init time there is only one userland executable running.

For executables from the userland/ directory such as userland/posix/count.c:

  • Shell 1:

    ./run --gdb-wait --kernel-cli 'init=/lkmc/posix/count.out'
  • Shell 2:

    ./run-gdb --userland userland/posix/count.c main

    Alternatively, we could also pass the full path to the executable:

    ./run-gdb --userland "$(./getvar userland_build_dir)/posix/count.out" main

    Path resolution is analogous to that of ./run --baremetal.

Then, as soon as boot ends, we are left inside a debug session that looks just like what gdbserver would produce.

3.6.2. GDB step debug userland BusyBox init

BusyBox custom init process:

  • Shell 1:

    ./run --gdb-wait --kernel-cli 'init=/bin/ls'
  • Shell 2:

    ./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

  • Shell 1:

    ./run --gdb-wait
  • Shell 2:

    ./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox init_main

init cannot be debugged with gdbserver without modifying the source, or else /sbin/init exits early with:

"must be run as PID 1"

3.6.3. GDB step debug userland non-init

Non-init process:

  • Shell 1:

    ./run --gdb-wait
  • Shell 2:

    ./run-gdb --userland userland/linux/rand_check.c main
  • Shell 1 after the boot finishes:

    ./linux/rand_check.out

This is the least reliable setup as there might be other processes that use the given virtual address.

3.6.3.1. GDB step debug userland non-init without --gdb-wait

TODO: if I try GDB step debug userland non-init without --gdb-wait and the break main that we do inside ./run-gdb says:

Cannot access memory at address 0x10604

and then GDB never breaks. Tested at ac8663a44a450c3eadafe14031186813f90c21e4 + 1.

The exact behaviour seems to depend on the architecture:

  • arm: happens always

  • x86_64: appears to happen only if you try to connect GDB as fast as possible, before init has been reached.

  • aarch64: could not observe the problem

We have also double checked the address with:

./run-toolchain --arch arm readelf -- \
  -s "$(./getvar --arch arm userland_build_dir)/linux/myinsmod.out" | \
  grep main

and from GDB:

info line main

and both give:

000105fc

which is just 8 bytes before 0x10604.

gdbserver also says 0x10604.

However, if do a Ctrl-C in GDB, and then a direct:

b *0x000105fc

it works. Why?!

On GEM5, x86 can also give the Cannot access memory at address, so maybe it is also unreliable on QEMU, and works just by coincidence.

3.7. GDB call

However this is failing for us:

  • some symbols are not visible to call even though b sees them

  • for those that are, call fails with an E14 error

E.g.: if we break on __x64_sys_write on count.sh:

>>> call printk(0, "asdf")
Could not fetch register "orig_rax"; remote failure reply 'E14'
>>> b printk
Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.
>>> call fdget_pos(fd)
No symbol "fdget_pos" in current context.
>>> b fdget_pos
Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)
>>>

even though fdget_pos is the first thing __x64_sys_write does:

581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
582         size_t, count)
583 {
584     struct fd f = fdget_pos(fd);

I also noticed that I get the same error:

Could not fetch register "orig_rax"; remote failure reply 'E14'

when trying to use:

fin

on many (all?) functions.

3.9. GDB step debug multicore userland

For a more minimal baremetal multicore setup, see: [arm-baremetal-multicore].

We can set and get which cores the Linux kernel allows a program to run on with sched_getaffinity and sched_setaffinity:

./run --cpus 2 --eval-after './linux/sched_getaffinity.out'

Sample output:

sched_getaffinity = 1 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

Which shows us that:

  • initially:

    • all 2 cores were enabled as shown by sched_getaffinity = 1 1

    • the process was randomly assigned to run on core 1 (the second one) as shown by sched_getcpu = 1. If we run this several times, it will also run on core 0 sometimes.

  • then we restrict the affinity to just core 0, and we see that the program was actually moved to core 0

The number of cores is modified as explained at: Section 24.3.1, “Number of cores”

taskset from the util-linux package sets the initial core affinity of a program:

./build-buildroot \
  --config 'BR2_PACKAGE_UTIL_LINUX=y' \
  --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
;
./run --eval-after 'taskset -c 1,1 ./linux/sched_getaffinity.out'

output:

sched_getaffinity = 0 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

so we see that the affinity was restricted to the second core from the start.

Let’s do a QEMU observation to justify this example being in the repository with userland breakpoints.

We will run our ./linux/sched_getaffinity.out infinitely many times, on core 0 and core 1 alternatively:

./run \
  --cpus 2 \
  --eval-after 'i=0; while true; do taskset -c $i,$i ./linux/sched_getaffinity.out; i=$((! $i)); done' \
  --gdb-wait \
;

on another shell:

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity.out" main

Then, inside GDB:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
  2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
(gdb) c
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
* 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
(gdb) c

and we observe that info threads shows the actual correct core on which the process was restricted to run by taskset!

TODO we then tried:

./run --cpus 2 --eval-after './linux/sched_getaffinity_threads.out'

and:

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity_threads.out"

to switch between two simultaneous live threads with different affinities, it just didn’t break on our threads:

b main_thread_0

Note that secondary cores in gem5 are kind of broken however: gem5 GDB step debug secondary cores.

Bibliography:

3.10. Linux kernel GDB scripts

We source the Linux kernel GDB scripts by default for lx-symbols, but they also contains some other goodies worth looking into.

Those scripts basically parse some in-kernel data structures to offer greater visibility with GDB.

All defined commands are prefixed by lx-, so to get a full list just try to tab complete that.

There aren’t as many as I’d like, and the ones that do exist are pretty self explanatory, but let’s give a few examples.

Show dmesg:

lx-dmesg
lx-cmdline

Dump the device tree to a fdtdump.dtb file in the current directory:

lx-fdtdump
pwd

List inserted kernel modules:

lx-lsmod

Sample output:

Address            Module                  Size  Used by
0xffffff80006d0000 hello                  16384  0

Bibliography:

3.10.1. lx-ps

List all processes:

lx-ps

Sample output:

0xffff88000ed08000 1 init
0xffff88000ed08ac0 2 kthreadd

The second and third fields are obviously PID and process name.

The first one is more interesting, and contains the address of the task_struct in memory.

This can be confirmed with:

p ((struct task_struct)*0xffff88000ed08000

which contains the correct PID for all threads I’ve tried:

pid = 1,

TODO get the PC of the kthreads: https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel Then we would be able to see where the threads are stopped in the code!

On ARM, I tried:

task_pt_regs((struct thread_info *)((struct task_struct)*0xffffffc00e8f8000))->uregs[ARM_pc]

but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb which are apparently not set?

Bibliography:

3.10.1.1. CONFIG_PID_IN_CONTEXTIDR

https://stackoverflow.com/questions/54133479/accessing-logical-software-thread-id-in-gem5 on ARM the kernel can store an indication of PID in the CONTEXTIDR_EL1 register, making that much easier to observe from simulators.

In particular, gem5 prints that number out by default on ExecAll messages!

./build-linux --arch aarch64 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --config 'CONFIG_PID_IN_CONTEXTIDR=y'
# Checkpoint run.
./run --arch aarch64 --emulator gem5 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --eval './gem5.sh'
# Trace run.
./run \
  --arch aarch64 \
  --emulator gem5 \
  --gem5-readfile 'posix/getpid.out; posix/getpid.out' \
  --gem5-restore 1 \
  --linux-build-id CONFIG_PID_IN_CONTEXTIDR \
  --trace FmtFlag,ExecAll,-ExecSymbol \
;

The terminal runs both programs which output their PID to stdout:

pid=44
pid=45

By quickly inspecting the trace.txt file, we immediately notice that the system.cpu: A<n> part of the logs, which used to always be system.cpu: A0, now has a few different values! Nice!

We can briefly summarize those values by removing repetitions:

cut -d' ' -f4 "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)" | uniq -c

gives:

  97227 A39
 147476 A38
 222052 A40
      1 terminal
1117724 A40
  27529 A31
  43868 A40
  27487 A31
 138349 A40
  13781 A38
 231246 A40
  25536 A38
  28337 A40
 214799 A38
 963561 A41
  92603 A38
  27511 A31
 224384 A38
 564949 A42
 182360 A38
 729009 A43
   8398 A23
  20200 A10
 636848 A43
 187995 A44
  27529 A31
  70071 A44
  16981 A0
 623806 A44
  16981 A0
 139319 A44
  24487 A0
 174986 A44
  25420 A0
  89611 A44
  16981 A0
 183184 A44
  24728 A0
  89608 A44
  17226 A0
 899075 A44
  24974 A0
 250608 A44
 137700 A43
1497997 A45
 227485 A43
 138147 A38
 482646 A46

I’m not smart enough to be able to deduce all of those IDs, but we can at least see that:

  • A44 and A45 are there as expected from stdout!

  • A39 must be the end of the execution of m5 checkpoint

  • so we guess that A38 is the shell as it comes next

  • the weird "terminal" line is 336969745500: system.terminal: attach terminal 0

  • which is the shell PID? I should have printed that as well :-)

  • why are there so many other PIDs? This was supposed to be a silent system without daemons!

  • A0 is presumably the kernel. However we see process switches without going into A0, so I’m not sure how, it appears to count kernel instructions as part of processes

  • A46 has to be the m5 exit call

Or if you want to have some real fun, try: baremetal/arch/aarch64/contextidr_el1.c:

./run --arch aarch64 --emulator gem5 --baremetal baremetal/arch/aarch64/contextidr_el1.c --trace-insts-stdout

in which we directly set the register ourselves! Output excerpt:

  31500: system.cpu: A0 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  32000: system.cpu: A1 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  32500: system.cpu: A1 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  33000: system.cpu: A1 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000002  flags=(IsInteger)
  33500: system.cpu: A1 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  34000: system.cpu: A1 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  34500: system.cpu: A1 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  35000: system.cpu: A1 T0 : @main+40    :   b.le   <main+12>         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  35500: system.cpu: A1 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  36000: system.cpu: A2 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000002  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  36500: system.cpu: A2 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  37000: system.cpu: A2 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000003  flags=(IsInteger)
  37500: system.cpu: A2 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  38000: system.cpu: A2 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  38500: system.cpu: A2 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  39000: system.cpu: A2 T0 : @main+40    :   b.le   <main+12>         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  39500: system.cpu: A2 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  40000: system.cpu: A3 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000003  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)

[armarm8-fa] D13.2.27 "CONTEXTIDR_EL1, Context ID Register (EL1)" documents CONTEXTIDR_EL1 as:

Identifies the current Process Identifier.

The value of the whole of this register is called the Context ID and is used by:

  • The debug logic, for Linked and Unlinked Context ID matching.

  • The trace logic, to identify the current process.

The significance of this register is for debug and trace use only.

Tested on 145769fc387dc5ee63ec82e55e6b131d9c968538 + 1.

3.11. Debug the GDB remote protocol

For when it breaks again, or you want to add a new feature!

./run --debug
./run-gdb --before '-ex "set remotetimeout 99999" -ex "set debug remote 1"' start_kernel

3.11.1. Remote 'g' packet reply is too long

This error means that the GDB server, e.g. in QEMU, sent more registers than the GDB client expected.

This can happen for the following reasons:

4. KGDB

KGDB is kernel dark magic that allows you to GDB the kernel on real hardware without any extra hardware support.

It is useless with QEMU since we already have full system visibility with -gdb. So the goal of this setup is just to prepare you for what to expect when you will be in the treches of real hardware.

KGDB is cheaper than JTAG (free) and easier to setup (all you need is serial), but with less visibility as it depends on the kernel working, so e.g.: dies on panic, does not see boot sequence.

First run the kernel with:

./run --kgdb

this passes the following options on the kernel CLI:

kgdbwait kgdboc=ttyS1,115200

kgdbwait tells the kernel to wait for KGDB to connect.

So the kernel sets things up enough for KGDB to start working, and then boot pauses waiting for connection:

<6>[    4.866050] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
<6>[    4.893205] 00:05: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
<6>[    4.916271] 00:06: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
<6>[    4.987771] KGDB: Registered I/O driver kgdboc
<2>[    4.996053] KGDB: Waiting for connection from remote gdb...

Entering kdb (current=0x(____ptrval____), pid 1) on processor 0 due to Keyboard Entry
[0]kdb>

KGDB expects the connection at ttyS1, our second serial port after ttyS0 which contains the terminal.

The last line is the KDB prompt, and is covered at: Section 4.3, “KDB”. Typing now shows nothing because that prompt is expecting input from ttyS1.

Instead, we connect to the serial port ttyS1 with GDB:

./run-gdb --kgdb --no-continue

Once GDB connects, it is left inside the function kgdb_breakpoint.

So now we can set breakpoints and continue as usual.

For example, in GDB:

continue

Then in QEMU:

./count.sh &
./kgdb.sh

rootfs_overlay/lkmc/kgdb.sh pauses the kernel for KGDB, and gives control back to GDB.

And now in GDB we do the usual:

break __x64_sys_write
continue
continue
continue
continue

And now you can count from KGDB!

If you do: break __x64_sys_write immediately after ./run-gdb --kgdb, it fails with KGDB: BP remove failed: <address>. I think this is because it would break too early on the boot sequence, and KGDB is not yet ready.

See also:

4.1. KGDB ARM

TODO: we would need a second serial for KGDB to work, but it is not currently supported on arm and aarch64 with -M virt that we use: https://unix.stackexchange.com/questions/479085/can-qemu-m-virt-on-arm-aarch64-have-multiple-serial-ttys-like-such-as-pl011-t/479340#479340

One possible workaround for this would be to use KDB ARM.

4.2. KGDB kernel modules

Just works as you would expect:

insmod timer.ko
./kgdb.sh

In GDB:

break lkmc_timer_callback
continue
continue
continue

and you now control the count.

4.3. KDB

KDB is a way to use KDB directly in your main console, without GDB.

Advantage over KGDB: you can do everything in one serial. This can actually be important if you only have one serial for both shell and .

Disadvantage: not as much functionality as GDB, especially when you use Python scripts. Notably, TODO confirm you can’t see the the kernel source code and line step as from GDB, since the kernel source is not available on guest (ah, if only debugging information supported full source, or if the kernel had a crazy mechanism to embed it).

Run QEMU as:

./run --kdb

This passes kgdboc=ttyS0 to the Linux CLI, therefore using our main console. Then QEMU:

[0]kdb> go

And now the kdb> prompt is responsive because it is listening to the main console.

After boot finishes, run the usual:

./count.sh &
./kgdb.sh

And you are back in KDB. Now you can count with:

[0]kdb> bp __x64_sys_write
[0]kdb> go
[0]kdb> go
[0]kdb> go
[0]kdb> go

And you will break whenever __x64_sys_write is hit.

You can get see further commands with:

[0]kdb> help

The other KDB commands allow you to step instructions, view memory, registers and some higher level kernel runtime data similar to the superior GDB Python scripts.

4.3.1. KDB graphic

You can also use KDB directly from the graphic window with:

./run --graphic --kdb

This setup could be used to debug the kernel on machines without serial, such as modern desktops.

This works because --graphics adds kbd (which stands for KeyBoarD!) to kgdboc.

4.3.2. KDB ARM

TODO neither arm and aarch64 are working as of 1cd1e58b023791606498ca509256cc48e95e4f5b + 1.

arm seems to place and hit the breakpoint correctly, but no matter how many go commands I do, the count.sh stdout simply does not show.

aarch64 seems to place the breakpoint correctly, but after the first go the kernel oopses with warning:

WARNING: CPU: 0 PID: 46 at /root/linux-kernel-module-cheat/submodules/linux/kernel/smp.c:416 smp_call_function_many+0xdc/0x358

and stack trace:

smp_call_function_many+0xdc/0x358
kick_all_cpus_sync+0x30/0x38
kgdb_flush_swbreak_addr+0x3c/0x48
dbg_deactivate_sw_breakpoints+0x7c/0xb8
kgdb_cpu_enter+0x284/0x6a8
kgdb_handle_exception+0x138/0x240
kgdb_brk_fn+0x2c/0x40
brk_handler+0x7c/0xc8
do_debug_exception+0xa4/0x1c0
el1_dbg+0x18/0x78
__arm64_sys_write+0x0/0x30
el0_svc_handler+0x74/0x90
el0_svc+0x8/0xc

My theory is that every serious ARM developer has JTAG, and no one ever tests this, and the kernel code is just broken.

5. gdbserver

Step debug userland processes to understand how they are talking to the kernel.

First build gdbserver into the root filesystem:

./build-buildroot --config 'BR2_PACKAGE_GDB=y'

Then on guest, to debug userland/linux/rand_check.c:

./gdbserver.sh ./c/command_line_arguments.out asdf qwer

And on host:

./run-gdb --gdbserver --userland userland/c/command_line_arguments.c main

or alternatively with the path to the executable itself:

./run --gdbserver --userland "$(./getvar userland_build_dir)/c/command_line_arguments.out"

5.1. gdbserver BusyBox

./gdbserver.sh ls

on host you need:

./run-gdb --gdbserver --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

5.2. gdbserver libc

Our setup gives you the rare opportunity to step debug libc and other system libraries.

For example in the guest:

./gdbserver.sh ./posix/count.out

Then on host:

./run-gdb --gdbserver --userland userland/posix/count.c main

and inside GDB:

break sleep
continue

And you are now left inside the sleep function of our default libc implementation uclibc libc/unistd/sleep.c!

You can also step into the sleep call:

step

This is made possible by the GDB command that we use by default:

set sysroot ${common_buildroot_build_dir}/staging

which automatically finds unstripped shared libraries on the host for us.

6. CPU architecture

The portability of the kernel and toolchains is amazing: change an option and most things magically work on completely different hardware.

To use arm instead of x86 for example:

./build-buildroot --arch arm
./run --arch arm

Debug:

./run --arch arm --gdb-wait
# On another terminal.
./run-gdb --arch arm

We also have one letter shorthand names for the architectures and --arch option:

# aarch64
./run -a A
# arm
./run -a a
# x86_64
./run -a x

Known quirks of the supported architectures are documented in this section.

6.1. x86_64

6.1.1. ring0

This example illustrates how reading from the x86 control registers with mov crX, rax can only be done from kernel land on ring0.

From kernel land:

insmod ring0.ko

works and output the registers, for example:

cr0 = 0xFFFF880080050033
cr2 = 0xFFFFFFFF006A0008
cr3 = 0xFFFFF0DCDC000

However if we try to do it from userland:

./ring0.out

stdout gives:

Segmentation fault

and dmesg outputs:

traps: ring0.out[55] general protection ip:40054c sp:7fffffffec20 error:0 in ring0.out[400000+1000]

Sources:

In both cases, we attempt to run the exact same code which is shared on the ring0.h header file.

Bibliography:

6.2. arm

6.2.1. Run arm executable in aarch64

I’ve tried:

./run-toolchain --arch aarch64 gcc -- -static ~/test/hello_world.c -o "$(./getvar p9_dir)/a.out"
./run --arch aarch64 --eval-after '/mnt/9p/data/a.out'

but it fails with:

a.out: line 1: syntax error: unexpected word (expecting ")")

6.3. MIPS

We used to "support" it until f8c0502bb2680f2dbe7c1f3d7958f60265347005 (it booted) but dropped since one was testing it often.

If you want to revive and maintain it, send a pull request.

6.4. Other architectures

It should not be too hard to port this repository to any architecture that Buildroot supports. Pull requests are welcome.

7. init

When the Linux kernel finishes booting, it runs an executable as the first and only userland process. This executable is called the init program.

The init process is then responsible for setting up the entire userland (or destroying everything when you want to have fun).

This typically means reading some configuration files (e.g. /etc/initrc) and forking a bunch of userland executables based on those files, including the very interactive shell that we end up on.

systemd provides a "popular" init implementation for desktop distros as of 2017.

BusyBox provides its own minimalistic init implementation which Buildroot, and therefore this repo, uses by default.

The init program can be either an executable shell text file, or a compiled ELF file. It becomes easy to accept this once you see that the exec system call handles both cases equally: https://unix.stackexchange.com/questions/174062/can-the-init-process-be-a-shell-script-in-linux/395375#395375

The init executable is searched for in a list of paths in the root filesystem, including /init, /sbin/init and a few others. For more details see: Section 7.3, “Path to init”

7.1. Replace init

To have more control over the system, you can replace BusyBox’s init with your own.

The most direct way to replace init with our own is to just use the init= command line parameter directly:

./run --kernel-cli 'init=/lkmc/count.sh'

This just counts every second forever and does not give you a shell.

This method is not very flexible however, as it is hard to reliably pass multiple commands and command line arguments to the init with it, as explained at: Section 7.4, “Init environment”.

For this reason, we have created a more robust helper method with the --eval option:

./run --eval 'echo "asdf qwer";insmod hello.ko;./linux/poweroff.out'

It is basically a shortcut for:

./run --kernel-cli 'init=/lkmc/eval_base64.sh - lkmc_eval="insmod hello.ko;./linux/poweroff.out"'

This allows quoting and newlines by base64 encoding on host, and decoding on guest, see: Section 17.3.1, “Kernel command line parameters escaping”.

It also automatically chooses between init= and rcinit= for you, see: Section 7.3, “Path to init”

--eval replaces BusyBox' init completely, which makes things more minimal, but also has has the following consequences:

  • /etc/fstab mounts are not done, notably /proc and /sys, test it out with:

    ./run --eval 'echo asdf;ls /proc;ls /sys;echo qwer'
  • no shell is launched at the end of boot for you to interact with the system. You could explicitly add a sh at the end of your commands however:

    ./run --eval 'echo hello;sh'

The best way to overcome those limitations is to use: Section 7.2, “Run command at the end of BusyBox init”

If the script is large, you can add it to a gitignored file and pass that to --eval as in:

echo '
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' > data/gitignore.sh
./run --eval "$(cat data/gitignore.sh)"

or add it to a file to the root filesystem guest and rebuild:

echo '#!/bin/sh
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' > rootfs_overlay/lkmc/gitignore.sh
chmod +x rootfs_overlay/lkmc/gitignore.sh
./build-buildroot
./run --kernel-cli 'init=/lkmc/gitignore.sh'

Remember that if your init returns, the kernel will panic, there are just two non-panic possibilities:

  • run forever in a loop or long sleep

  • poweroff the machine

7.1.1. poweroff.out

Just using BusyBox' poweroff at the end of the init does not work and the kernel panics:

./run --eval poweroff

because BusyBox' poweroff tries to do some fancy stuff like killing init, likely to allow userland to shutdown nicely.

But this fails when we are init itself!

BusyBox' poweroff works more brutally and effectively if you add -f:

./run --eval 'poweroff -f'

but why not just use our minimal ./linux/poweroff.out and be done with it?

./run --eval './linux/poweroff.out'

7.1.2. sleep_forever.out

I dare you to guess what this does:

./run --eval './posix/sleep_forever.out'

This executable is a convenient simple init that does not panic and sleeps instead.

7.1.3. time_boot.out

Get a reasonable answer to "how long does boot take in guest time?":

./run --eval-after './linux/time_boot.c'

That executable writes to dmesg directly through /dev/kmsg a message of type:

[    2.188242] /path/to/linux-kernel-module-cheat/userland/linux/time_boot.c

which tells us that boot took 2.188242 seconds based on the dmesg timestamp.

7.2. Run command at the end of BusyBox init

Use the --eval-after option is for you rely on something that BusyBox' init set up for you like /etc/fstab:

./run --eval-after 'echo asdf;ls /proc;ls /sys;echo qwer'

After the commands run, you are left on an interactive shell.

The above command is basically equivalent to:

./run --kernel-cli-after-dash 'lkmc_eval="insmod hello.ko;./linux/poweroff.out;"'

where the lkmc_eval option gets evaled by our default rootfs_overlay/etc/init.d/S98 startup script.

Except that --eval-after is smarter and uses base64 encoding.

Alternatively, you can also add the comamdns to run to a new init.d entry to run at the end o the BusyBox init:

cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore
vim rootfs_overlay/etc/init.d/S99.gitignore
./build-buildroot
./run

and they will be run automatically before the login prompt.

Scripts under /etc/init.d are run by /etc/init.d/rcS, which gets called by the line ::sysinit:/etc/init.d/rcS in /etc/inittab.

7.3. Path to init

The init is selected at:

  • initrd or initramfs system: /init, a custom one can be set with the rdinit= kernel command line parameter

  • otherwise: default is /sbin/init, followed by some other paths, a custom one can be set with init=

The final init that actually got selected is shown on Linux v5.9.2 a line of type:

<6>[    0.309984] Run /sbin/init as init process

at the very end of the boot logs.

7.4. Init environment

The kernel parses parameters from the kernel command line up to "-"; if it doesn’t recognize a parameter and it doesn’t contain a '.', the parameter gets passed to init: parameters with '=' go into init’s environment, others are passed as command line arguments to init. Everything after "-" is passed as an argument to init.

And you can try it out with:

./run --kernel-cli 'init=/lkmc/linux/init_env_poweroff.out' --kernel-cli-after-dash 'asdf=qwer zxcv'

From the generated QEMU command, we see that the kernel CLI at LKMC 69f5745d3df11d5c741551009df86ea6c61a09cf now contains:

init=/lkmc/linux/init_env_poweroff.out console=ttyS0 - lkmc_home=/lkmc asdf=qwer zxcv

and the init program outputs:

args:
/lkmc/linux/init_env_poweroff.out
-
zxcv

env:
HOME=/
TERM=linux
lkmc_home=/lkmc
asdf=qwer

As of the Linux kernel v5.7 (possibly earlier, I’ve skipped a few releases), boot also shows the init arguments and environment very clearly, which is a great addition:

<6>[    0.309984] Run /sbin/init as init process
<7>[    0.309991]   with arguments:
<7>[    0.309997]     /sbin/init
<7>[    0.310004]     nokaslr
<7>[    0.310010]     -
<7>[    0.310016]   with environment:
<7>[    0.310022]     HOME=/
<7>[    0.310028]     TERM=linux
<7>[    0.310035]     earlyprintk=pl011,0x1c090000
<7>[    0.310041]     lkmc_home=/lkmc

7.4.1. init arguments

The annoying dash - gets passed as a parameter to init, which makes it impossible to use this method for most non custom executables.

Arguments with dots that come after - are still treated specially (of the form subsystem.somevalue) and disappear, from args, e.g.:

./run --kernel-cli 'init=/lkmc/linux/init_env_poweroff.out' --kernel-cli-after-dash '/lkmc/linux/poweroff.out'

outputs:

args
/lkmc/linux/init_env_poweroff.out
-
ab

so see how a.b is gone.

The simple workaround is to just create a shell script that does it, e.g. as we’ve done at: rootfs_overlay/lkmc/gem5_exit.sh.

7.4.2. init environment env

Wait, where do HOME and TERM come from? (greps the kernel). Ah, OK, the kernel sets those by default: https://github.com/torvalds/linux/blob/94710cac0ef4ee177a63b5227664b38c95bbf703/init/main.c#L173

const char *envp_init[MAX_INIT_ENVS+2] = { "HOME=/", "TERM=linux", NULL, };

7.4.3. BusyBox shell init environment

On top of the Linux kernel, the BusyBox /bin/sh shell will also define other variables.

We can explore the shenanigans that the shell adds on top of the Linux kernel with:

./run --kernel-cli 'init=/bin/sh'

From there we observe that:

env

gives:

SHLVL=1
HOME=/
TERM=linux
PWD=/

therefore adding SHLVL and PWD to the default kernel exported variables.

Furthermore, to increase confusion, if you list all non-exported shell variables https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values with:

set

then it shows more variables, notably:

PATH='/sbin:/usr/sbin:/bin:/usr/bin'
7.4.3.1. BusyBox shell initrc files

Login shells source some default files, notably:

/etc/profile
$HOME/.profile

We provide /.profile from rootfs_overlay/.profile, and use the default BusyBox /etc/profile.

The shell knows that it is a login shell if the first character of argv[0] is -, see also: https://stackoverflow.com/questions/2050961/is-argv0-name-of-executable-an-accepted-standard-or-just-a-common-conventi/42291142#42291142

When we use just init=/bin/sh, the Linux kernel sets argv[0] to /bin/sh, which does not start with -.

However, if you use ::respawn:-/bin/sh on inttab described at TTY, BusyBox' init sets argv[0][0] to -, and so does getty. This can be observed with:

cat /proc/$$/cmdline

8. initrd

The kernel can boot from an CPIO file, which is a directory serialization format much like tar: https://superuser.com/questions/343915/tar-vs-cpio-what-is-the-difference

The bootloader, which for us is provided by QEMU itself, is then configured to put that CPIO into memory, and tell the kernel that it is there.

This is very similar to the kernel image itself, which already gets put into memory by the QEMU -kernel option.

With this setup, you don’t even need to give a root filesystem to the kernel: it just does everything in memory in a ramfs.

To enable initrd instead of the default ext2 disk image, do:

./build-buildroot --initrd
./run --initrd

By looking at the QEMU run command generated, you can see that we didn’t give the -drive option at all:

cat "$(./getvar run_dir)/run.sh"

Instead, we used the QEMU -initrd option to point to the .cpio filesystem that Buildroot generated for us.

Try removing that -initrd option to watch the kernel panic without rootfs at the end of boot.

When using .cpio, there can be no filesystem persistency across boots, since all file operations happen in memory in a tmpfs:

date >f
poweroff
cat f
# can't open 'f': No such file or directory

which can be good for automated tests, as it ensures that you are using a pristine unmodified system image every time.

Not however that we already disable disk persistency by default on ext2 filesystems even without --initrd: Section 23.3, “Disk persistency”.

One downside of this method is that it has to put the entire filesystem into memory, and could lead to a panic:

end Kernel panic - not syncing: Out of memory and no killable processes...

This can be solved by increasing the memory as explained at Memory size:

./run --initrd --memory 256M

The main ingredients to get initrd working are:

TODO: how does the bootloader inform the kernel where to find initrd? https://unix.stackexchange.com/questions/89923/how-does-linux-load-the-initrd-image

8.1. initrd in desktop distros

Most modern desktop distributions have an initrd in their root disk to do early setup.

The rationale for this is described at: https://en.wikipedia.org/wiki/Initial_ramdisk

One obvious use case is having an encrypted root filesystem: you keep the initrd in an unencrypted partition, and then setup decryption from there.

I think GRUB then knows read common disk formats, and then loads that initrd to memory with a /boot/grub/grub.cfg directive of type:

initrd /initrd.img-4.4.0-108-generic

8.2. initramfs

initramfs is just like initrd, but you also glue the image directly to the kernel image itself using the kernel’s build system.

Try it out with:

./build-buildroot --initramfs
./build-linux --initramfs
./run --initramfs

Notice how we had to rebuild the Linux kernel this time around as well after Buildroot, since in that build we will be gluing the CPIO to the kernel image.

Now, once again, if we look at the QEMU run command generated, we see all that QEMU needs is the -kernel option, no -drive not even -initrd! Pretty cool:

cat "$(./getvar run_dir)/run.sh"

It is also interesting to observe how this increases the size of the kernel image if you do a:

ls -lh "$(./getvar linux_image)"

before and after using initramfs, since the .cpio is now glued to the kernel image.

Don’t forget that to stop using initramfs, you must rebuild the kernel without --initramfs to get rid of the attached CPIO image:

./build-linux
./run

Alternatively, consider using [linux-kernel-build-variants] if you need to switch between initramfs and non initramfs often:

./build-buildroot --initramfs
./build-linux --initramfs --linux-build-id initramfs
./run --initramfs --linux-build-id

Setting up initramfs is very easy: our scripts just set CONFIG_INITRAMFS_SOURCE to point to the CPIO path.

8.3. rootfs

This is how /proc/mounts shows the root filesystem:

  • hard disk: /dev/root on / type ext2 (rw,relatime,block_validity,barrier,user_xattr). That file does not exist however.

  • initrd: rootfs on / type rootfs (rw)

  • initramfs: rootfs on / type rootfs (rw)

TODO: understand /dev/root better:

8.4. gem5 initrd

This would require gem5 to load the CPIO into memory, just like QEMU. Grepping initrd shows some ARM hits under:

src/arch/arm/linux/atag.hh

but they are commented out.

8.5. gem5 initramfs

This could in theory be easier to make work than initrd since the emulator does not have to do anything special.

However, it didn’t: boot fails at the end because it does not see the initramfs, but rather tries to open our dummy root filesystem, which unsurprisingly does not have a format in a way that the kernel understands:

VFS: Cannot open root device "sda" or unknown-block(8,0): error -5

We think that this might be because gem5 boots directly vmlinux, and not from the final compressed images that contain the attached rootfs such as bzImage, which is what QEMU does, see also: Section 17.20.1, “vmlinux vs bzImage vs zImage vs Image”.

To do this failed test, we automatically pass a dummy disk image as of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91 since the scripts don’t handle a missing --disk-image well, much like is currently done for [baremetal].

Interestingly, using initramfs significantly slows down the gem5 boot, even though it did not work. For example, we’ve observed a 4x slowdown of as 17062a2e8b6e7888a14c3506e9415989362c58bf for aarch64. This must be because expanding the large attached CPIO must be expensive. We can clearly see from the kernel logs that the kernel just hangs at a point after the message PCI: CLS 0 bytes, default 64 for a long time before proceeding further.

9. Device tree

The device tree is a Linux kernel defined data structure that serves to inform the kernel how the hardware is setup.

Device trees serve to reduce the need for hardware vendors to patch the kernel: they just provide a device tree file instead, which is much simpler.

x86 does not use it device trees, but many other archs to, notably ARM.

This is notably because ARM boards:

  • typically don’t have discoverable hardware extensions like PCI, but rather just put everything on an SoC with magic register addresses

  • are made by a wide variety of vendors due to ARM’s licensing business model, which increases variability

The Linux kernel itself has several device trees under ./arch/<arch>/boot/dts, see also: https://stackoverflow.com/questions/21670967/how-to-compile-dts-linux-device-tree-source-files-to-dtb/42839737#42839737

9.1. DTB files

Files that contain device trees have the .dtb extension when compiled, and .dts when in text form.

You can convert between those formats with:

"$(./getvar buildroot_host_dir)"/bin/dtc -I dtb -O dts -o a.dts a.dtb
"$(./getvar buildroot_host_dir)"/bin/dtc -I dts -O dtb -o a.dtb a.dts

Buildroot builds the tool due to BR2_PACKAGE_HOST_DTC=y.

On Ubuntu 18.04, the package is named:

sudo apt-get install device-tree-compiler

Device tree files are provided to the emulator just like the root filesystem and the Linux kernel image.

In real hardware, those components are also often provided separately. For example, on the Raspberry Pi 2, the SD card must contain two partitions:

  • the first contains all magic files, including the Linux kernel and the device tree

  • the second contains the root filesystem

9.2. Device tree syntax

Good format descriptions:

Minimal example

/dts-v1/;

/ {
    a;
};

Check correctness with:

dtc a.dts

Separate nodes are simply merged by node path, e.g.:

/dts-v1/;

/ {
    a;
};

/ {
    b;
};

then dtc a.dts gives:

/dts-v1/;

/ {
        a;
        b;
};

9.3. Get device tree from a running kernel

This is specially interesting because QEMU and gem5 are capable of generating DTBs that match the selected machine depending on dynamic command line parameters for some types of machines.

So observing the device tree from the guest allows to easily see what the emulator has generated.

Compile the dtc tool into the root filesystem:

./build-buildroot \
  --arch aarch64 \
  --config 'BR2_PACKAGE_DTC=y' \
  --config 'BR2_PACKAGE_DTC_PROGRAMS=y' \
;

-M virt for example, which we use by default for aarch64, boots just fine without the -dtb option:

./run --arch aarch64

Then, from inside the guest:

dtc -I fs -O dts /sys/firmware/devicetree/base

contains:

        cpus {
                #address-cells = <0x1>;
                #size-cells = <0x0>;

                cpu@0 {
                        compatible = "arm,cortex-a57";
                        device_type = "cpu";
                        reg = <0x0>;
                };
        };

9.4. Device tree emulator generation

Since emulators know everything about the hardware, they can automatically generate device trees for us, which is very convenient.

This is the case for both QEMU and gem5.

For example, if we increase the number of cores to 2:

./run --arch aarch64 --cpus 2

QEMU automatically adds a second CPU to the DTB!

                cpu@0 {
                cpu@1 {

The action seems to be happening at: hw/arm/virt.c.

You can dump the DTB QEMU generated with:

./run --arch aarch64 -- -machine dumpdtb=dtb.dtb

gem5 fs_bigLITTLE 2a9573f5942b5416fb0570cf5cb6cdecba733392 can also generate its own DTB.

gem5 can generate DTBs on ARM with --generate-dtb. The generated DTB is placed in the m5out directory named as system.dtb.

10. KVM

KVM is Linux kernel interface that greatly speeds up execution of virtual machines.

You can make QEMU or gem5 by passing enabling KVM with:

./run --kvm

KVM works by running userland instructions natively directly on the real hardware instead of running a software simulation of those instructions.

Therefore, KVM only works if you the host architecture is the same as the guest architecture. This means that this will likely only work for x86 guests since almost all development machines are x86 nowadays. Unless you are running an ARM desktop for some weird reason :-)

We don’t enable KVM by default because:

  • it limits visibility, since more things are running natively:

  • QEMU kernel boots are already fast enough for most purposes without it

One important use case for KVM is to fast forward gem5 execution, often to skip boot, take a gem5 checkpoint, and then move on to a more detailed and slow simulation

10.1. KVM arm

TODO: we haven’t gotten it to work yet, but it should be doable, and this is an outline of how to do it. Just don’t expect this to tested very often for now.

We can test KVM on arm by running this repository inside an Ubuntu arm QEMU VM.

This produces no speedup of course, since the VM is already slow since it cannot use KVM on the x86 host.

Then, from inside that image:

sudo apt-get install git
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
./setup -y

and then proceed exactly as in Prebuilt setup.

We don’t want to build the full Buildroot image inside the VM as that would be way too slow, thus the recommendation for the prebuilt setup.

TODO: do the right thing and cross compile QEMU and gem5. gem5’s Python parts might be a pain. QEMU should be easy: https://stackoverflow.com/questions/26514252/cross-compile-qemu-for-arm

10.2. gem5 KVM

While gem5 does have KVM, as of 2019 its support has not been very good, because debugging it is harder and people haven’t focused intensively on it.

X86 was broken with pending patches: https://www.mail-archive.com/[email protected]/msg15046.html It failed immediately on:

panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)

also mentioned at:

Bibliography:

11. User mode simulation

Both QEMU and gem5 have an user mode simulation mode in addition to full system simulation that we consider elsewhere in this project.

In QEMU, it is called just "user mode", and in gem5 it is called syscall emulation mode.

In both, the basic idea is the same.

User mode simulation takes regular userland executables of any arch as input and executes them directly, without booting a kernel.

Instead of simulating the full system, it translates normal instructions like in full system mode, but magically forwards system calls to the host OS.

Advantages over full system simulation:

  • the simulation may run faster since you don’t have to simulate the Linux kernel and several device models

  • you don’t need to build your own kernel or root filesystem, which saves time. You still need a toolchain however, but the pre-packaged ones may work fine.

Disadvantages:

  • lower guest to host portability:

    • TODO confirm: host OS == guest OS?

    • TODO confirm: the host Linux kernel should be newer than the kernel the executable was built for.

      It may still work even if that is not the case, but could fail is a missing system call is reached.

      The target Linux kernel of the executable is a GCC toolchain build-time configuration.

    • emulator implementers have to keep up with libc changes, some of which break even a C hello world due setup code executed before main.

  • cannot be used to test the Linux kernel or any devices, and results are less representative of a real system since we are faking more

11.1. QEMU user mode getting started

Let’s run userland/c/command_line_arguments.c built with the Buildroot toolchain on QEMU user mode:

./build user-mode-qemu
./run \
  --userland userland/c/command_line_arguments.c \
  --cli-args='asdf "qw er"' \
;

Output:

/path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/command_line_arguments.out
asdf
qw er

./run --userland path resolution is analogous to that of ./run --baremetal.

./build user-mode-qemu first builds Buildroot, and then runs ./build-userland, which is further documented at: Section 2.8, “Userland setup”. It also builds QEMU. If you ahve already done a QEMU Buildroot setup previously, this will be very fast.

If you modify the userland programs, rebuild simply with:

./build-userland

To rebuild just QEMU userland if you hack it, use:

./build-qemu --mode userland

The:

--mode userland

is needed because QEMU has two separate executables:

  • qemu-x86_64 for userland

  • qemu-system-x86_64 for full system

11.1.1. User mode GDB

It’s nice when the obvious just works, right?

./run \
  --arch aarch64 \
  --gdb-wait \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

and on another shell:

./run-gdb \
  --arch aarch64 \
  --userland userland/c/command_line_arguments.c \
  main \
;

Or alternatively, if you are using tmux, do everything in one go with:

./run \
  --arch aarch64 \
  --gdb \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

To stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: [freestanding-programs].

11.2. User mode tests

Automatically run all userland tests that can be run in user mode simulation, and check that they exit with status 0:

./build --all-archs test-executables-userland
./test-executables --all-archs --all-emulators

Or just for QEMU:

./build --all-archs test-executables-userland-qemu
./test-executables --all-archs --emulator qemu

This script skips a manually configured list of tests, notably:

  • tests that depend on a full running kernel and cannot be run in user mode simulation, e.g. those that rely on kernel modules

  • tests that require user interaction

  • tests that take perceptible amounts of time

  • known bugs we didn’t have time to fix ;-)

Tests under userland/libs/ are only run if --package or --package-all are given as described at [userland-libs-directory].

The gem5 tests require building statically with build id static, see also: Section 11.7, “gem5 syscall emulation mode”. TODO automate this better.

See: [test-this-repo] for more useful testing tips.

11.3. User mode Buildroot executables

If you followed QEMU Buildroot setup, you can now run the executables created by Buildroot directly as:

./run \
  --userland "$(./getvar buildroot_target_dir)/bin/echo" \
  --cli-args='asdf' \
;

To easily explore the userland executable environment interactively, you can do:

./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --terminal \
;

or:

./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --cli-args='-c "uname -a && pwd"' \
;

Here is an interesting examples of this: Section 17.19.1, “Linux Test Project”

11.4. User mode simulation with glibc

At 125d14805f769104f93c510bedaa685a52ec025d we moved Buildroot from uClibc to glibc, and caused some user mode pain, which we document here.

11.4.1. FATAL: kernel too old failure in userland simulation

glibc has a check for kernel version, likely obtained from the uname syscall, and if the kernel is not new enough, it quits.

Both gem5 and QEMU however allow setting the reported uname version from the command line for User mode simulation, which we do to always match our toolchain.

QEMU by default copies the host uname value, but we always override it in our scripts.

Determining the right number to use for the kernel version is of course highly non-trivial and would require an extensive userland test suite, which most emulators don’t have.

./run --arch aarch64 --kernel-version 4.18 --userland userland/posix/uname.c

The QEMU source that does this is at: https://github.com/qemu/qemu/blob/v3.1.0/linux-user/syscall.c#L8931 The default ID is just hardcoded on the source.

Bibliography:

11.4.2. stack smashing detected when using glibc

For some reason QEMU / glibc x86_64 picks up the host libc, which breaks things.

Other archs work as they different host libc is skipped. User mode static executables also work.

We have worked around this with with https://bugs.launchpad.net/qemu/+bug/1701798/comments/12 from the thread: https://bugs.launchpad.net/qemu/+bug/1701798 by creating the file: rootfs_overlay/etc/ld.so.cache which is a symlink to a file that cannot exist: /dev/null/nonexistent.

Reproduction:

rm -f "$(./getvar buildroot_target_dir)/etc/ld.so.cache"
./run --userland userland/c/hello.c
./run --userland userland/c/hello.c --qemu-which host

Outcome:

*** stack smashing detected ***: <unknown> terminated
qemu: uncaught target signal 6 (Aborted) - core dumped

To get things working again, restore ld.so.cache with:

./build-buildroot

I’ve also tested on an Ubuntu 16.04 guest and the failure is different one:

qemu: uncaught target signal 4 (Illegal instruction) - core dumped

A non-QEMU-specific example of stack smashing is shown at: https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264

Tested at: 2e32389ebf1bedd89c682aa7b8fe42c3c0cf96e5 + 1.

11.5. User mode static executables

Example:

./build-userland \
  --arch aarch64 \
  --static \
;
./run \
  --arch aarch64 \
  --static \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

Running dynamically linked executables in QEMU requires pointing it to the root filesystem with the -L option so that it can find the dynamic linker and shared libraries, see also:

We pass -L by default, so everything just works.

However, in case something goes wrong, you can also try statically linked executables, since this mechanism tends to be a bit more stable, for example:

Running statically linked executables sometimes makes things break:

11.5.1. User mode static executables with dynamic libraries

One limitation of static executables is that Buildroot mostly only builds dynamic versions of libraries (the libc is an exception).

So programs that rely on those libraries might not compile as GCC can’t find the .a version of the library.

For example, if we try to build [blas] statically:

./build-userland --package openblas --static -- userland/libs/openblas/hello.c

it fails with:

ld: cannot find -lopenblas
11.5.1.1. C++ static and pthreads

g++ and pthreads also causes issues:

As a consequence, the following just hangs as of LKMC ca0403849e03844a328029d70c08556155dc1cd0 + 1 the example userland/cpp/atomic/std_atomic.cpp:

./run --userland userland/cpp/atomic/std_atomic.cpp --static

And before that, it used to fail with other randomly different errors, e.g.:

qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.
qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.

And a native Ubuntu 18.04 AMD64 run with static compilation segfaults.

As of LKMC f5d4998ff51a548ed3f5153aacb0411d22022058 the aarch64 error:

./run --arch aarch64 --userland userland/cpp/atomic/fail.cpp --static

is:

terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error 16781344
qemu: uncaught target signal 6 (Aborted) - core dumped

The workaround:

-pthread -Wl,--whole-archive -lpthread -Wl,--no-whole-archive

fixes some of the problems, but not all TODO which were missing?, so we are just skipping those tests for now.

11.6. syscall emulation mode program stdin

The following work on both QEMU and gem5 as of LKMC 99d6bc6bc19d4c7f62b172643be95d9c43c26145 + 1. Interactive input:

./run --userland userland/c/getchar.c

A line of type should show:

enter a character:

and after pressing say a and Enter, we get:

you entered: a

Note however that due to QEMU user mode does not show stdout immediately we don’t really see the initial enter a character line.

Non-interactive input from a file by forwarding emulators stdin implicitly through our Python scripts:

printf a > f.tmp
./run --userland userland/c/getchar.c < f.tmp

Input from a file by explicitly requesting our scripts to use it via the Python API:

printf a > f.tmp
./run --emulator gem5 --userland userland/c/getchar.c --stdin-file f.tmp

This is especially useful when running tests that require stdin input.

11.7. gem5 syscall emulation mode

Less robust than QEMU’s, but still usable:

There are much more unimplemented syscalls in gem5 than in QEMU. Many of those are trivial to implement however.

So let’s just play with some static ones:

./build-userland --arch aarch64
./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

TODO: how to escape spaces on the command line arguments?

GDB step debug also works normally on gem5:

./run \
  --arch aarch64 \
  --emulator gem5 \
  --gdb-wait \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;
./run-gdb \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  main \
;

11.7.2. gem5 syscall emulation exit status

As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy se.py script does not forward the exit status of syscall emulation mode, you can test it with:

./run --dry-run --emulator gem5 --userland userland/c/false.c

Then manually run the generated gem5 CLI, and do:

echo $?

and the output is always 0.

Instead, it just outputs a message to stdout just like for m5 fail:

Simulated exit code not 0! Exit code is 1

which we parse in run and then exit with the correct result ourselves…​

11.7.3. gem5 syscall emulation mode syscall tracing

Since gem5 has to implement syscalls itself in syscall emulation mode, it can of course clearly see which syscalls are being made, and we can log them for debug purposes with gem5 tracing, e.g.:

./run \
  --emulator gem5 \
  --userland userland/arch/x86_64/freestanding/linux/hello.S \
  --trace-stdout \
  --trace ExecAll,SyscallBase,SyscallVerbose \
;

the trace as of f2eeceb1cde13a5ff740727526bf916b356cee38 + 1 contains:

      0: system.cpu A0 T0 : @asm_main_after_prologue    : mov   rdi, 0x1
      0: system.cpu A0 T0 : @asm_main_after_prologue.0  :   MOV_R_I : limm   rax, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7    : mov rdi, 0x1
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7.0  :   MOV_R_I : limm   rdi, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14    : lea        rsi, DS:[rip + 0x19]
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14.0  :   LEA_R_P : rdip   t7, %ctrl153,  : IntAlu :  D=0x000000000040008d  flags=(IsInteger|IsMicroop|IsDelayedCommit|IsFirstMicroop)
   2500: system.cpu A0 T0 : @asm_main_after_prologue+14.1  :   LEA_R_P : lea   rsi, DS:[t7 + 0x19] : IntAlu :  D=0x00000000004000a6  flags=(IsInteger|IsMicroop|IsLastMicroop)
   3500: system.cpu A0 T0 : @asm_main_after_prologue+21    : mov        rdi, 0x6
   3500: system.cpu A0 T0 : @asm_main_after_prologue+21.0  :   MOV_R_I : limm   rdx, 0x6 : IntAlu :  D=0x0000000000000006  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   4000: system.cpu: T0 : syscall write called w/arguments 1, 4194470, 6, 0, 0, 0
hello
   4000: system.cpu: T0 : syscall write returns 6
   4000: system.cpu A0 T0 : @asm_main_after_prologue+28    :   syscall    eax           : IntAlu :   flags=(IsInteger|IsSerializeAfter|IsNonSpeculative|IsSyscall)
   5000: system.cpu A0 T0 : @asm_main_after_prologue+30    : mov        rdi, 0x3c
   5000: system.cpu A0 T0 : @asm_main_after_prologue+30.0  :   MOV_R_I : limm   rax, 0x3c : IntAlu :  D=0x000000000000003c  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   6000: system.cpu A0 T0 : @asm_main_after_prologue+37    : mov        rdi, 0
   6000: system.cpu A0 T0 : @asm_main_after_prologue+37.0  :   MOV_R_I : limm   rdi, 0  : IntAlu :  D=0x0000000000000000  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   6500: system.cpu: T0 : syscall exit called w/arguments 0, 4194470, 6, 0, 0, 0
   6500: system.cpu: T0 : syscall exit returns 0
   6500: system.cpu A0 T0 : @asm_main_after_prologue+44    :   syscall    eax           : IntAlu :   flags=(IsInteger|IsSerializeAfter|IsNonSpeculative|IsSyscall)

so we see that two syscall lines were added for each syscall, showing the syscall inputs and exit status, just like a mini strace!

11.7.4. gem5 syscall emulation multithreading

gem5 user mode multithreading has been particularly flaky compared to QEMU’s, but work is being put into improving it.

In gem5 syscall simulation, the fork syscall checks if there is a free CPU, and if there is a free one, the new threads runs on that CPU.

Otherwise, the fork call, and therefore higher level interfaces to fork such as pthread_create also fail and return a failure return status in the guest.

For example, if we use just one CPU for userland/posix/pthread_self.c which spawns one thread besides main:

./run --cpus 1 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args 1

fails with this error message coming from the guest stderr:

pthread_create: Resource temporarily unavailable

It works however if we add on extra CPU:

./run --cpus 2 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args 1

Once threads exit, their CPU is freed and becomes available for new fork calls: For example, the following run spawns a thread, joins it, and then spawns again, and 2 CPUs are enough:

./run --cpus 2 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args '1 2'

because at each point in time, only up to two threads are running.

gem5 syscall emulation does show the expected number of cores when queried, e.g.:

./run --cpus 1 --userland userland/cpp/thread_hardware_concurrency.cpp --emulator gem5
./run --cpus 2 --userland userland/cpp/thread_hardware_concurrency.cpp --emulator gem5

outputs 1 and 2 respectively.

This can also be clearly by running sched_getcpu:

./run \
  --arch aarch64 \
  --cli-args  4 \
  --cpus 8 \
  --emulator gem5 \
  --userland userland/linux/sched_getcpu.c \
;

which necessarily produces an output containing the CPU numbers from 1 to 4 and no higher:

1
3
4
2

TODO why does the 2 come at the end here? Would be good to do a detailed assembly run analysis.

11.7.5. gem5 syscall emulation multiple executables

gem5 syscall emulation has the nice feature of allowing you to run multiple executables "at once".

Each executable starts running on the next free core much as if it had been forked right at the start of simulation: gem5 syscall emulation multithreading.

This can be useful to quickly create deterministic multi-CPU workload.

se.py --cmd takes a semicolon separated list, so we could do which LKMC exposes this by taking --userland multiple times as in:

./run \
  --arch aarch64 \
  --cpus 2 \
  --emulator gem5 \
  --userland userland/posix/getpid.c \
  --userland userland/posix/getpid.c \
;

We need at least one CPU per executable, just like when forking new processes.

The outcome of this is that we see two different pid messages printed to stdout:

pid=101
pid=100

since from [gem5-process] we can see that se.py sets up one different PID per executable starting at 100:

    workloads = options.cmd.split(';')
    idx = 0
    for wrkld in workloads:
        process = Process(pid = 100 + idx)

We can also see that these processes are running concurrently with gem5 tracing by hacking:

  --debug-flags ExecAll \
  --debug-file cout \

which starts with:

      0: system.cpu1: A0 T0 : @__end__+274873647040    :   add   x0, sp, #0         : IntAlu :  D=0x0000007ffffefde0  flags=(IsInteger)
      0: system.cpu0: A0 T0 : @__end__+274873647040    :   add   x0, sp, #0         : IntAlu :  D=0x0000007ffffefde0  flags=(IsInteger)
    500: system.cpu0: A0 T0 : @__end__+274873647044    :   bl   <__end__+274873649648> : IntAlu :  D=0x0000004000001008  flags=(IsInteger|IsControl|IsDirectControl|IsUncondControl|IsCall)
    500: system.cpu1: A0 T0 : @__end__+274873647044    :   bl   <__end__+274873649648> : IntAlu :  D=0x0000004000001008  flags=(IsInteger|IsControl|IsDirectControl|IsUncondControl|IsCall)

and therefore shows one instruction running on each CPU for each process at the same time.

11.7.5.1. gem5 syscall emulation --smt

gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4 syscall emulation has an --smt option presumably for [hardware-threads] but it has been neglected forever it seems: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/104

If we start from the manually hacked working command from gem5 syscall emulation multiple executables and try to add:

--cpu 1 --cpu-type Derivo3CPU --caches

We choose DerivO3CPU because of the se.py assert:

example/se.py:115:        assert(options.cpu_type == "DerivO3CPU")

But then that fails with:

gem5.opt: /path/to/linux-kernel-module-cheat/out/gem5/master3/build/ARM/cpu/o3/cpu.cc:205: FullO3CPU<Impl>::FullO3CPU(DerivO3CPUParams*) [with Impl = O3CPUImpl]: Assertion `params->numPhysVecPredRegs >= numThreads * TheISA::NumVecPredRegs' failed.
Program aborted at tick 0

11.8. QEMU user mode quirks

11.8.1. QEMU user mode does not show stdout immediately

At 8d8307ac0710164701f6e14c99a69ee172ccbb70 + 1, I noticed that if you run userland/posix/count.c:

./run --userland userland/posix/count_to.c --cli-args 3

it first waits for 3 seconds, then the program exits, and then it dumps all the stdout at once, instead of counting once every second as expected.

The same can be reproduced by copying the raw QEMU command and piping it through tee, so I don’t think it is a bug in our setup:

/path/to/linux-kernel-module-cheat/out/qemu/default/x86_64-linux-user/qemu-x86_64 \
  -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \
  /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/posix/count.out \
  3 \
| tee

TODO: investigate further and then possibly post on QEMU mailing list.

11.8.1.1. QEMU user mode does not show errors

Similarly to QEMU user mode does not show stdout immediately, QEMU error messages do not show at all through pipes.

In particular, it does not say anything if you pass it a non-existing executable:

qemu-x86_64 asdf | cat

So we just check ourselves manually

12. Kernel module utilities

12.1. insmod

./run --eval-after 'insmod hello.ko'

12.2. myinsmod

If you are feeling raw, you can insert and remove modules with our own minimal module inserter and remover!

# init_module
./linux/myinsmod.out hello.ko
# finit_module
./linux/myinsmod.out hello.ko "" 1
./linux/myrmmod.out hello

which teaches you how it is done from C code.

Source:

The Linux kernel offers two system calls for module insertion:

  • init_module

  • finit_module

and:

man init_module

documents that:

The finit_module() system call is like init_module(), but reads the module to be loaded from the file descriptor fd. It is useful when the authenticity of a kernel module can be determined from its location in the filesystem; in cases where that is possible, the overhead of using cryptographically signed modules to determine the authenticity of a module can be avoided. The param_values argument is as for init_module().

finit is newer and was added only in v3.8. More rationale: https://lwn.net/Articles/519010/

12.3. modprobe

modprobe searches for modules installed under:

ls /lib/modules/<kernel_version>

and specified in the modules.order file.

This is the default install path for CONFIG_SOME_MOD=m modules built with make modules_install in the Linux kernel tree, with root path given by INSTALL_MOD_PATH, and therefore canonical in that sense.

Currently, there are only two kinds of kernel modules that you can try out with modprobe:

We are not installing out custom ./build-modules modules there, because:

12.4. kmod

The more "reference" kernel.org implementation of lsmod, insmod, rmmod, etc.: https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git

Default implementation on desktop distros such as Ubuntu 16.04, where e.g.:

ls -l /bin/lsmod

gives:

lrwxrwxrwx 1 root root 4 Jul 25 15:35 /bin/lsmod -> kmod

and:

dpkg -l | grep -Ei

contains:

ii  kmod                                        22-1ubuntu5                                         amd64        tools for managing Linux kernel modules

BusyBox also implements its own version of those executables, see e.g. modprobe. Here we will only describe features that differ from kmod to the BusyBox implementation.

12.4.1. module-init-tools

Name of a predecessor set of tools.

12.4.2. kmod modprobe

kmod’s modprobe can also load modules under different names to avoid conflicts, e.g.:

sudo modprobe vmhgfs -o vm_hgfs

13. Filesystems

13.1. OverlayFS

OverlayFS is a filesystem merged in the Linux kernel in 3.18.

As the name suggests, OverlayFS allows you to merge multiple directories into one. The following minimal runnable examples should give you an intuition on how it works:

We are very interested in this filesystem because we are looking for a way to make host cross compiled executables appear on the guest root / without reboot.

This would have several advantages:

  • makes it faster to test modified guest programs

    • not rebooting is fundamental for gem5, where the reboot is very costly.

    • no need to regenerate the root filesystem at all and reboot

    • overcomes the check_bin_arch problem as shown at: [rpath]

  • we could keep the base root filesystem very small, which implies:

    • less host disk usage, no need to copy the entire ./getvar out_rootfs_overlay_dir to the image again

    • no need to worry about [br2-target-rootfs-ext2-size]

We can already make host files appear on the guest with 9P, but they appear on a subdirectory instead of the root.

If they would appear on the root instead, that would be even more awesome, because you would just use the exact same paths relative to the root transparently.

For example, we wouldn’t have to mess around with variables such as PATH and LD_LIBRARY_PATH.

The idea is to:

We already have a prototype of this running from fstab on guest at /mnt/overlay, but it has the following shortcomings:

  • changes to underlying filesystems are not visible on the overlay unless you remount with mount -r remount /mnt/overlay, as mentioned on the kernel docs:

    Changes to the underlying filesystems while part of a mounted overlay
    filesystem are not allowed.  If the underlying filesystem is changed,
    the behavior of the overlay is undefined, though it will not result in
    a crash or deadlock.

    This makes everything very inconvenient if you are inside chroot action. You would have to leave chroot, remount, then come back.

  • the overlay does not contain sub-filesystems, e.g. /proc. We would have to re-mount them. But should be doable with some automation.

Even more awesome than chroot would be to pivot_root, but I couldn’t get that working either:

13.2. Secondary disk

A simpler and possibly less overhead alternative to 9P would be to generate a secondary disk image with the benchmark you want to rebuild.

Then you can umount and re-mount on guest without reboot.

To build the secondary disk image run build-disk2:

./build-disk2

This will put the entire [out-rootfs-overlay-dir] into a squashfs filesystem.

Then, if that filesystem is present, ./run will automatically pass it as the second disk on the command line.

For example, from inside QEMU, you can mount that disk with:

mkdir /mnt/vdb
mount /dev/vdb /mnt/vdb
/mnt/vdb/lkmc/c/hello.out

To update the secondary disk while a simulation is running to avoid rebooting, first unmount in the guest:

umount /mnt/vdb

and then on the host:

# Edit the file.
vim userland/c/hello.c
./build-userland
./build-disk2

and now you can re-run the updated version of the executable on the guest after remounting it.

14. Graphics

Both QEMU and gem5 are capable of outputting graphics to the screen, and taking mouse and keyboard input.

14.1. QEMU text mode

Text mode is the default mode for QEMU.

The opposite of text mode is QEMU graphic mode

In text mode, we just show the serial console directly on the current terminal, without opening a QEMU GUI window.

You cannot see any graphics from text mode, but text operations in this mode, including:

making this a good default, unless you really need to use with graphics.

Text mode works by sending the terminal character by character to a serial device.

This is different from a display screen, where each character is a bunch of pixels, and it would be much harder to convert that into actual terminal text.

For more details, see:

Note that you can still see an image even in text mode with the VNC:

./run --vnc

and on another terminal:

./vnc

but there is not terminal on the VNC window, just the CONFIG_LOGO penguin.

14.1.1. Quit QEMU from text mode

However, our QEMU setup captures Ctrl + C and other common signals and sends them to the guest, which makes it hard to quit QEMU for the first time since there is no GUI either.

The simplest way to quit QEMU, is to do:

Ctrl-A X

Alternative methods include:

14.2. QEMU graphic mode

Enable graphic mode with:

./run --graphic

Outcome: you see a penguin due to CONFIG_LOGO.

For a more exciting GUI experience, see: Section 14.4, “X11 Buildroot”

Text mode is the default due to the following considerable advantages:

  • copy and paste commands and stdout output to / from host

  • get full panic traces when you start making the kernel crash :-) See also: https://unix.stackexchange.com/questions/208260/how-to-scroll-up-after-a-kernel-panic

  • have a large scroll buffer, and be able to search it, e.g. by using tmux on host

  • one less window floating around to think about in addition to your shell :-)

  • graphics mode has only been properly tested on x86_64.

Text mode has the following limitations over graphics mode:

  • you can’t see graphics such as those produced by X11 Buildroot

  • very early kernel messages such as early console in extract_kernel only show on the GUI, since at such early stages, not even the serial has been setup.

x86_64 has a VGA device enabled by default, as can be seen as:

./qemu-monitor info qtree

and the Linux kernel picks it up through the fbdev graphics system as can be seen from:

cat /dev/urandom > /dev/fb0

14.2.2. QEMU Graphic mode arm

14.2.2.1. QEMU graphic mode arm terminal

TODO: on arm, we see the penguin and some boot messages, but don’t get a shell at then end:

./run --arch aarch64 --graphic

I think it does not work because the graphic window is DRM only, i.e.:

cat /dev/urandom > /dev/fb0

fails with:

cat: write error: No space left on device

and has no effect, and the Linux kernel does not appear to have a built-in DRM console as it does for fbdev with fbcon.

There is however one out-of-tree implementation: kmscon.

14.2.2.2. QEMU graphic mode arm terminal implementation

arm and aarch64 rely on the QEMU CLI option:

-device virtio-gpu-pci

and the kernel config options:

CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y

Unlike x86, arm and aarch64 don’t have a display device attached by default, thus the need for virtio-gpu-pci.

See also https://wiki.qemu.org/Documentation/Platforms/ARM (recently edited and corrected by yours truly…​ :-)).

14.2.2.3. QEMU graphic mode arm VGA
-device VGA
# We use virtio-gpu because the legacy VGA framebuffer is
# very troublesome on aarch64, and virtio-gpu is the only
# video device that doesn't implement it.

so maybe it is not possible?

14.3. gem5 graphic mode

gem5 does not have a "text mode", since it cannot redirect the Linux terminal to same host terminal where the executable is running: you are always forced to connect to the terminal with gem-shell.

TODO could not get it working on x86_64, only ARM.

More concretely, first build the kernel with the gem5 arm Linux kernel patches, and then run:

./build-linux \
  --arch arm \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
./run --arch arm --emulator gem5 --linux-build-id gem5-v4.15

and then on another shell:

vinagre localhost:5900

The CONFIG_LOGO penguin only appears after several seconds, together with kernel messages of type:

[    0.152755] [drm] found ARM HDLCD version r0p0
[    0.152790] hdlcd 2b000000.hdlcd: bound virt-encoder (ops 0x80935f94)
[    0.152795] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.152799] [drm] No driver support for vblank timestamp query.
[    0.215179] Console: switching to colour frame buffer device 240x67
[    0.230389] hdlcd 2b000000.hdlcd: fb0:  frame buffer device
[    0.230509] [drm] Initialized hdlcd 1.0.0 20151021 for 2b000000.hdlcd on minor 0

The port 5900 is incremented by one if you already have something running on that port, gem5 stdout tells us the right port on stdout as:

system.vncserver: Listening for connections on port 5900

and when we connect it shows a message:

info: VNC client attached

Alternatively, you can also dump each new frame to an image file with --frame-capture:

./run \
  --arch arm \
  --emulator gem5 \
  --linux-build-id gem5-v4.15 \
  -- --frame-capture \
;

This creates on compressed PNG whenever the screen image changes inside the m5out directory with filename of type:

frames_system.vncserver/fb.<frame-index>.<timestamp>.png.gz

It is fun to see how we get one new frame whenever the white underscore cursor appears and reappears under the penguin!

The last frame is always available uncompressed at: system.framebuffer.png.

TODO kmscube failed on aarch64 with:

kmscube[706]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x92000006, in libgbm.so.1.0.0[7fbf6a6000+e000]

14.3.1. Graphic mode gem5 aarch64

For aarch64 we also need to configure the kernel with linux_config/display:

git -C "$(./getvar linux_source_dir)" fetch https://gem5.googlesource.com/arm/linux gem5/v4.15:gem5/v4.15
git -C "$(./getvar linux_source_dir)" checkout gem5/v4.15
./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/display \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
git -C "$(./getvar linux_source_dir)" checkout -
./run --arch aarch64 --emulator gem5 --linux-build-id gem5-v4.15

This is because the gem5 aarch64 defconfig does not enable HDLCD like the 32 bit one arm one for some reason.

14.3.2. gem5 graphic mode DP650

TODO get working. There is an unmerged patchset at: https://gem5-review.googlesource.com/c/public/gem5/+/11036/1

The DP650 is a newer display hardware than HDLCD. TODO is its interface publicly documented anywhere? Since it has a gem5 model and in-tree Linux kernel support, that information cannot be secret?

The key option to enable support in Linux is DRM_MALI_DISPLAY=y which we enable at linux_config/display.

Build the kernel exactly as for Graphic mode gem5 aarch64 and then run with:

./run --arch aarch64 --dp650 --emulator gem5 --linux-build-id gem5-v4.15

14.3.3. gem5 graphic mode internals

We cannot use mainline Linux because the gem5 arm Linux kernel patches are required at least to provide the CONFIG_DRM_VIRT_ENCODER option.

gem5 emulates the HDLCD ARM Holdings hardware for arm and aarch64.

The kernel uses HDLCD to implement the DRM interface, the required kernel config options are present at: linux_config/display.

TODO: minimize out the --custom-config-file. If we just remove it on arm: it does not work with a failing dmesg:

[    0.066208] [drm] found ARM HDLCD version r0p0
[    0.066241] hdlcd 2b000000.hdlcd: bound virt-encoder (ops drm_vencoder_ops)
[    0.066247] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.066252] [drm] No driver support for vblank timestamp query.
[    0.066276] hdlcd 2b000000.hdlcd: Cannot do DMA to address 0x0000000000000000
[    0.066281] swiotlb: coherent allocation failed for device 2b000000.hdlcd size=8294400
[    0.066288] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0 #1
[    0.066293] Hardware name: V2P-AARCH64 (DT)
[    0.066296] Call trace:
[    0.066301]  dump_backtrace+0x0/0x1b0
[    0.066306]  show_stack+0x24/0x30
[    0.066311]  dump_stack+0xb8/0xf0
[    0.066316]  swiotlb_alloc_coherent+0x17c/0x190
[    0.066321]  __dma_alloc+0x68/0x160
[    0.066325]  drm_gem_cma_create+0x98/0x120
[    0.066330]  drm_fbdev_cma_create+0x74/0x2e0
[    0.066335]  __drm_fb_helper_initial_config_and_unlock+0x1d8/0x3a0
[    0.066341]  drm_fb_helper_initial_config+0x4c/0x58
[    0.066347]  drm_fbdev_cma_init_with_funcs+0x98/0x148
[    0.066352]  drm_fbdev_cma_init+0x40/0x50
[    0.066357]  hdlcd_drm_bind+0x220/0x428
[    0.066362]  try_to_bring_up_master+0x21c/0x2b8
[    0.066367]  component_master_add_with_match+0xa8/0xf0
[    0.066372]  hdlcd_probe+0x60/0x78
[    0.066377]  platform_drv_probe+0x60/0xc8
[    0.066382]  driver_probe_device+0x30c/0x478
[    0.066388]  __driver_attach+0x10c/0x128
[    0.066393]  bus_for_each_dev+0x70/0xb0
[    0.066398]  driver_attach+0x30/0x40
[    0.066402]  bus_add_driver+0x1d0/0x298
[    0.066408]  driver_register+0x68/0x100
[    0.066413]  __platform_driver_register+0x54/0x60
[    0.066418]  hdlcd_platform_driver_init+0x20/0x28
[    0.066424]  do_one_initcall+0x44/0x130
[    0.066428]  kernel_init_freeable+0x13c/0x1d8
[    0.066433]  kernel_init+0x18/0x108
[    0.066438]  ret_from_fork+0x10/0x1c
[    0.066444] hdlcd 2b000000.hdlcd: Failed to set initial hw configuration.
[    0.066470] hdlcd 2b000000.hdlcd: master bind failed: -12
[    0.066477] hdlcd: probe of 2b000000.hdlcd failed with error -12

So what other options are missing from gem5_defconfig? It would be cool to minimize it out to better understand the options.

14.4. X11 Buildroot

Once you’ve seen the CONFIG_LOGO penguin as a sanity check, you can try to go for a cooler X11 Buildroot setup.

Build and run:

./build-buildroot --config-fragment buildroot_config/x11
./run --graphic

Inside QEMU:

startx

And then from the GUI you can start exciting graphical programs such as:

xcalc
xeyes
x11
Figure 1. X11 Buildroot graphical user interface screenshot

We don’t build X11 by default because it takes a considerable amount of time (about 20%), and is not expected to be used by most users: you need to pass the -x flag to enable it.

Not sure how well that graphics stack represents real systems, but if it does it would be a good way to understand how it works.

To x11 packages have an xserver prefix as in:

./build-buildroot --config-fragment buildroot_config/x11 -- xserver_xorg-server-reconfigure

the easiest way to find them out is to just list "$(./getvar buildroot_build_build_dir)/x*.

TODO as of: c2696c978d6ca88e8b8599c92b1beeda80eb62b2 I noticed that startx leads to a BUG_ON:

[    2.809104] WARNING: CPU: 0 PID: 51 at drivers/gpu/drm/ttm/ttm_bo_vm.c:304 ttm_bo_vm_open+0x37/0x40

14.4.1. X11 Buildroot mouse not moving

TODO 9076c1d9bcc13b6efdb8ef502274f846d8d4e6a1 I’m 100% sure that it was working before, but I didn’t run it forever, and it stopped working at some point. Needs bisection, on whatever commit last touched x11 stuff.

-show-cursor did not help, I just get to see the host cursor, but the guest cursor still does not move.

Doing:

watch -n 1 grep i8042 /proc/interrupts

shows that interrupts do happen when mouse and keyboard presses are done, so I expect that it is some wrong either with:

  • QEMU. Same behaviour if I try the host’s QEMU 2.10.1 however.

  • X11 configuration. We do have BR2_PACKAGE_XDRIVER_XF86_INPUT_MOUSE=y.

/var/log/Xorg.0.log contains the following interesting lines:

[    27.549] (II) LoadModule: "mouse"
[    27.549] (II) Loading /usr/lib/xorg/modules/input/mouse_drv.so
[    27.590] (EE) <default pointer>: Cannot find which device to use.
[    27.590] (EE) <default pointer>: cannot open input device
[    27.590] (EE) PreInit returned 2 for "<default pointer>"
[    27.590] (II) UnloadModule: "mouse"

The file /dev/inputs/mice does not exist.

Note that our current link:kernel_confi_fragment sets:

# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set

for gem5, so you might want to remove those lines to debug this.

14.4.2. X11 Buildroot ARM

On ARM, startx hangs at a message:

vgaarb: this pci device is not a vga device

and nothing shows on the screen, and:

grep EE /var/log/Xorg.0.log

says:

(EE) Failed to load module "modesetting" (module does not exist, 0)

A friend told me this but I haven’t tried it yet:

  • xf86-video-modesetting is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things.

  • xf86-video-fbdev should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the Xorg.conf

15. Networking

15.1. Enable networking

We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable as explained at: [resource-tradeoff-guidelines]

To enable networking on Buildroot, simply run:

ifup -a

That command goes over all (-a) the interfaces in /etc/network/interfaces and brings them up.

Then test it with:

wget google.com
cat index.html

Disable networking with:

ifdown -a

To enable networking by default after boot, use the methods documented at Run command at the end of BusyBox init.

15.2. ping

ping does not work within QEMU by default, e.g.:

ping google.com

hangs after printing the header:

PING google.com (216.58.204.46): 56 data bytes

15.3. Guest host networking

In this section we discuss how to interact between the guest and the host through networking.

First ensure that you can access the external network since that is easier to get working, see: Section 15, “Networking”.

15.3.1. Host to guest networking

15.3.1.1. nc host to guest

With nc we can create the most minimal example possible as a sanity check.

On guest run:

nc -l -p 45455

Then on host run:

echo asdf | nc localhost 45455

asdf appears on the guest.

This uses:

  • BusyBox' nc utility, which is enabled with CONFIG_NC=y

  • nc from the netcat-openbsd package on an Ubuntu 18.04 host

Only this specific port works by default since we have forwarded it on the QEMU command line.

We us this exact procedure to connect to gdbserver.

15.3.1.2. ssh into guest

Not enabled by default due to the build / runtime overhead. To enable, build with:

./build-buildroot --config 'BR2_PACKAGE_OPENSSH=y'

Then inside the guest turn on sshd:

./sshd.sh

And finally on host:

ssh root@localhost -p 45456
15.3.1.3. gem5 host to guest networking

Could not do port forwarding from host to guest, and therefore could not use gdbserver: https://stackoverflow.com/questions/48941494/how-to-do-port-forwarding-from-guest-to-host-in-gem5

15.3.2. Guest to host networking

Then in the host, start a server:

python -m SimpleHTTPServer 8000

And then in the guest, find the IP we need to hit with:

ip rounte

which gives:

default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 scope link  src 10.0.2.15

so we use in the guest:

wget 10.0.2.2:8000

Bibliography:

15.4. 9P

The 9p protocol allows the guest to mount a host directory.

Both QEMU and gem5 9P support 9P.

15.4.1. 9P vs NFS

All of 9P and NFS (and sshfs) allow sharing directories between guest and host.

Advantages of 9P

  • requires sudo on the host to mount

  • we could share a guest directory to the host, but this would require running a server on the guest, which adds simulation overhead

    Furthermore, this would be inconvenient, since what we usually want to do is to share host cross built files with the guest, and to do that we would have to copy the files over after the guest starts the server.

  • QEMU implements 9P natively, which makes it very stable and convenient, and must mean it is a simpler protocol than NFS as one would expect.

    This is not the case for gem5 7bfb7f3a43f382eb49853f47b140bfd6caad0fb8 unfortunately, which relies on the diod host daemon, although it is not unfeasible that future versions could implement it natively as well.

Advantages of NFS:

  • way more widely used and therefore stable and available, not to mention that it also works on real hardware.

  • the name does not start with a digit, which is an invalid identifier in all programming languages known to man. Who in their right mind would call a software project as such? It does not even match the natural order of Plan 9; Plan then 9: P9!

15.4.2. 9P getting started

As usual, we have already set everything up for you. On host:

cd "$(./getvar p9_dir)"
uname -a > host

Guest:

cd /mnt/9p/data
cat host
uname -a > guest

Host:

cat guest

The main ingredients for this are:

Bibliography:

15.4.3. gem5 9P

Is possible on aarch64 as shown at: https://gem5-review.googlesource.com/c/public/gem5/+/22831, and it is just a matter of exposing to X86 for those that want it.

Enable it by passing the --vio-9p option on the fs.py gem5 command line:

./run --arch aarch64 --emulator gem5 -- --vio-9p

Then on the guest:

mkdir -p /mnt/9p/gem5
mount -t 9p -o trans=virtio,version=9p2000.L,aname=/path/to/linux-kernel-module-cheat/out/run/gem5/aarch64/0/m5out/9p/share gem5 /mnt/9p/gem5
echo asdf > /mnt/9p/gem5/qwer

Yes, you have to pass the full path to the directory on the host. Yes, this is horrible.

The shared directory is:

out/run/gem5/aarch64/0/m5out/9p/share

so we can observe the file the guest wrote from the host with:

out/run/gem5/aarch64/0/m5out/9p/share/qwer

and vice versa:

echo zxvc > out/run/gem5/aarch64/0/m5out/9p/share/qwer

is now visible from the guest:

cat /mnt/9p/gem5/qwer

Checkpoint restore with an open mount will likely fail because gem5 uses an ugly external executable to implement diod. The protocol is not very complex, and QEMU implements it in-tree, which is what gem5 should do as well at some point.

Also checkpoint without --vio-9p and restore with --vio-9p did not work either, the mount fails.

However, this did work, on guest:

unmount /mnt/9p/gem5
m5 checkpoint

then restore with the detalied CPU of interest e.g.

./run --arch aarch64 --emulator gem5 -- --vio-9p --cpu-type DerivO3CPU --caches

Tested on gem5 b2847f43c91e27f43bd4ac08abd528efcf00f2fd, LKMC 52a5fdd7c1d6eadc5900fc76e128995d4849aada.

15.4.4. NFS

TODO: get working.

9P is better with emulation, but let’s just get this working for fun.

First make sure that this works: Section 15.3.2, “Guest to host networking”.

Then, build the kernel with NFS support:

./build-linux --config-fragment linux_config/nfs

Now on host:

sudo apt-get install nfs-kernel-server

Now edit /etc/exports to contain:

/tmp *(rw,sync,no_root_squash,no_subtree_check)

and restart the server:

sudo systemctl restart nfs-kernel-server

Now on guest:

mkdir /mnt/nfs
mount -t nfs 10.0.2.2:/tmp /mnt/nfs

TODO: failing with:

mount: mounting 10.0.2.2:/tmp on /mnt/nfs failed: No such device

And now the /tmp directory from host is not mounted on guest!

If you don’t want to start the NFS server after the next boot automatically so save resources, do:

systemctl disable nfs-kernel-server

17. Linux kernel

17.1. Linux kernel configuration

17.1.1. Modify kernel config

To modify a single option on top of our default kernel configs, do:

./build-linux --config 'CONFIG_FORTIFY_SOURCE=y'

Kernel modules depend on certain kernel configs, and therefore in general you might have to clean and rebuild the kernel modules after changing the kernel config:

./build-modules --clean
./build-modules

and then proceed as in Your first kernel module hack.

You might often get way without rebuilding the kernel modules however.

To use an extra kernel config fragment file on top of our defaults, do:

printf '
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
' > data/myconfig
./build-linux --config-fragment 'data/myconfig'

To use just your own exact .config instead of our defaults ones, use:

./build-linux --custom-config-file data/myconfig

There is also a shortcut --custom-config-file-gem5 to use the gem5 arm Linux kernel patches.

The following options can all be used together, sorted by decreasing config setting power precedence:

  • --config

  • --config-fragment

  • --custom-config-file

To do a clean menu config yourself and use that for the build, do:

./build-linux --clean
./build-linux --custom-config-target menuconfig

But remember that every new build re-configures the kernel by default, so to keep your configs you will need to use on further builds:

./build-linux --no-configure

So what you likely want to do instead is to save that as a new defconfig and use it later as:

./build-linux --no-configure --no-modules-install savedefconfig
cp "$(./getvar linux_build_dir)/defconfig" data/myconfig
./build-linux --custom-config-file data/myconfig

You can also use other config generating targets such as defconfig with the same method as shown at: Section 17.1.3.1.1, “Linux kernel defconfig”.

17.1.2. Find the kernel config

Get the build config in guest:

zcat /proc/config.gz

or with our shortcut:

./conf.sh

or to conveniently grep for a specific option case insensitively:

./conf.sh ikconfig

This is enabled by:

CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y

From host:

cat "$(./getvar linux_config)"
./linux/scripts/extract-ikconfig "$(./getvar vmlinux)"

although this can be useful when someone gives you a random image.

17.1.3. About our Linux kernel configs

By default, build-linux generates a .config that is a mixture of:

To find out which kernel configs are being used exactly, simply run:

./build-linux --dry-run

and look for the merge_config.sh call. This script from the Linux kernel tree, as the name suggests, merges multiple configuration files into one as explained at: https://unix.stackexchange.com/questions/224887/how-to-script-make-menuconfig-to-automate-linux-kernel-build-configuration/450407#450407

For each arch, the base of our configs are named as:

linux_config/buildroot-<arch>

These configs are extracted directly from a Buildroot build with update-buildroot-kernel-configs.

Note that Buildroot can sed override some of the configurations, e.g. it forces CONFIG_BLK_DEV_INITRD=y when BR2_TARGET_ROOTFS_CPIO is on. For this reason, those configs are not simply copy pasted from Buildroot files, but rather from a Buildroot kernel build, and then minimized with make savedefconfig: https://stackoverflow.com/questions/27899104/how-to-create-a-defconfig-file-from-a-config

On top of those, we add the following by default:

17.1.3.1. About Buildroot’s kernel configs

To see Buildroot’s base configs, start from buildroot/configs/qemu_x86_64_defconfig.

That file contains BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-4.15.config", which points to the base config file used: board/qemu/x86_64/linux-4.15.config.

arm, on the other hand, uses buildroot/configs/qemu_arm_vexpress_defconfig, which contains BR2_LINUX_KERNEL_DEFCONFIG="vexpress", and therefore just does a make vexpress_defconfig, and gets its config from the Linux kernel tree itself.

17.1.3.1.1. Linux kernel defconfig

To boot defconfig from disk on Linux and see a shell, all we need is these missing virtio options:

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
  --config CONFIG_VIRTIO_PCI=y \
  --config CONFIG_VIRTIO_BLK=y \
;
./run --linux-build-id defconfig

Oh, and check this out:

du -h \
  "$(./getvar vmlinux)" \
  "$(./getvar --linux-build-id defconfig vmlinux)" \
;

Output:

360M    /path/to/linux-kernel-module-cheat/out/linux/default/x86_64/vmlinux
47M     /path/to/linux-kernel-module-cheat/out/linux/defconfig/x86_64/vmlinux

Brutal. Where did we go wrong?

The extra virtio options are not needed if we use initrd:

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run --initrd --linux-build-id defconfig

On aarch64, we can boot from initrd with:

./build-linux \
  --arch aarch64 \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run \
  --arch aarch64 \
  --initrd \
  --linux-build-id defconfig \
  --memory 2G \
;

We need the 2G of memory because the CPIO is 600MiB due to a humongous amount of loadable kernel modules!

In aarch64, the size situation is inverted from x86_64, and this can be seen on the vmlinux size as well:

118M    /path/to/linux-kernel-module-cheat/out/linux/default/aarch64/vmlinux
240M    /path/to/linux-kernel-module-cheat/out/linux/defconfig/aarch64/vmlinux

So it seems that the ARM devs decided rather than creating a minimal config that boots QEMU, to try and make a single config that boots every board in existence. Terrible!

Tested on 1e2b7f1e5e9e3073863dc17e25b2455c8ebdeadd + 1.

17.1.3.1.2. Linux kernel min config

linux_config/min contains minimal tweaks required to boot gem5 or for using our slightly different QEMU command line options than Buildroot on all archs.

It is one of the default config fragments we use, as explained at: Section 17.1.3, “About our Linux kernel configs”>.

Having the same config working for both QEMU and gem5 (oh, the hours of bisection) means that you can deal with functional matters in QEMU, which runs much faster, and switch to gem5 only for performance issues.

We can build just with min on top of the base config with:

./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/min \
  --custom-config-file linux_config/buildroot-aarch64 \
  --linux-build-id min \
;

vmlinux had a very similar size to the default. It seems that linux_config/buildroot-aarch64 contains or implies most linux_config/default options already? TODO: that seems odd, really?

Tested on 649d06d6758cefd080d04dc47fd6a5a26a620874 + 1.

17.1.3.2. Notable alternate gem5 kernel configs

Other configs which we had previously tested at 4e0d9af81fcce2ce4e777cb82a1990d7c2ca7c1e are:

17.2. Kernel version

17.2.1. Find the kernel version

We try to use the latest possible kernel major release version.

In QEMU:

cat /proc/version

or in the source:

cd "$(./getvar linux_source_dir)"
git log | grep -E '    Linux [0-9]+\.' | head

17.2.2. Update the Linux kernel

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: [bisection].

17.2.2.1. Update the Linux kernel LKMC procedure

First, use use the branching procedure described at: [update-a-forked-submodule]

Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: [test-this-repo]. The only tests that can be skipped are essentially the [baremetal] tests.

Before comitting, don’t forget to update:

  • the linux_kernel_version constant in common.py

  • the tagline of this repository on:

    • this README

    • the GitHub project description

17.2.3. Downgrade the Linux kernel

The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

The default Linux kernel version is bumped in Buildroot with commit messages of type:

linux: bump default to version 4.9.6

So you can try:

git log --grep 'linux: bump default to version'

Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.

You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.

Finally, after downgrading Buildroot, if something does not work, you might also have to make some changes to how this repo uses Buildroot, as the Buildroot configuration options might have changed.

We don’t expect those changes to be very difficult. A good way to approach the task is to:

  • do a dry run build to get the equivalent Bash commands used:

    ./build-buildroot --dry-run
  • build the Buildroot documentation for the version you are going to use, and check if all Buildroot build commands make sense there

Then, if you spot an option that is wrong, some grepping in this repo should quickly point you to the code you need to modify.

It also possible that you will need to apply some patches from newer Buildroot versions for it to build, due to incompatibilities with the host Ubuntu packages and that Buildroot version. Just read the error message, and try:

  • git log master — packages/<pkg>

  • Google the error message for mailing list hits

Successful port reports:

17.3. Kernel command line parameters

Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the execve system call does to userland processes.

This allows us to control the behaviour of the kernel without rebuilding anything.

With QEMU, QEMU itself acts as the bootloader, and provides the -append option and we expose it through ./run --kernel-cli, e.g.:

./run --kernel-cli 'foo bar'

Then inside the host, you can check which options were given with:

cat /proc/cmdline

They are also printed at the beginning of the boot message:

dmesg | grep "Command line"

See also:

The arguments are documented in the kernel documentation: https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html

When dealing with real boards, extra command line options are provided on some magic bootloader configuration file, e.g.:

17.3.1. Kernel command line parameters escaping

Double quotes can be used to escape spaces as in opt="a b", but double quotes themselves cannot be escaped, e.g. opt"a\"b"

This even lead us to use base64 encoding with --eval!