系列文章 : simulation / emulation 學習筆記

在前一篇文章，我們可以在 gem5 上 boot 一個 ARM 的 linux kernel，這裡開始學習這個 python configuration 裡面的內容。

大綱

• class Named
  • gem5/src/base/named.hh
• class Group
  • gem5/src/base/stats/group.hh
• class Drainable
  • gem5/src/sim/drain.hh
• class Serializable
  • gem5/src/sim/serialize.hh
• class EventQueue
  • gem5/src/sim/eventq.hh
• class EventManager
  • gem5/src/sim/eventq.hh
• class SimObject
  • gem5/src/sim/sim_object.hh
  • gem5/src/python/m5/SimObject.py

class Named

{ https://github.com/gem5/gem5/blob/develop/src/base/named.hh }

• private
  • const std::string _name;
    • 儲存這個 object 的 name
• public
  • Named(std::string_view name_) : _name(name_)
    • 建構子
    • 有趣的是參數的型態為 std::string_view，這讓參數可以是 string literal ( e.g. “hello”, “abc” )，而不用預先轉型成 std::string。
  • virtual std::string name()
    • 這個 method 會回傳該 object 的 name
    • 因為是 virtual 的，所以 derived class 可以自己實作自己的 name，來回傳 derived class 所預期的 name。
    • 這個 function 主要的用途是會被 DPRINTF ( https://github.com/gem5/gem5/blob/stable/src/base/trace.hh#L209 ) 所使用，用以顯示該 object 的 name。
• class Named 也是 class SimObject 的 base class。而幾乎所有 gem5 內的 simulation components ( e.g. CPUs, caches, memory controllers ) 的 base class 都有 SimObject。這讓 gem5 可以很簡單的讓每一個 simulation components 都有著自己的名字，這樣在 configuration, statistics, debugging 都很方便。

class Group

{ https://github.com/gem5/gem5/blob/stable/src/base/stats/group.hh }

• 這個 class 跟計算效能數據有關，可以等之後需要處理效能數據，再來仔細研究這個 class
• 這也是 class SimObject 的 base class 之一。
• 當我們想要在 stats.txt 多輸出一項效能數據的時候，可以使用 ADD_STAT。例如下面的例子，我們可以在 stats.txt 裡面看到 simSeconds，這項數據代表我們模擬了幾秒。並不是真實世界的秒數，而是虛擬世界的秒數。例如真實世界跑了 5 分鐘的模擬，說不定在虛擬世界才過了 1 秒。

   // src/sim/root.cc
   ADD_STAT(simSeconds, statistics::units::Second::get(),
             "Number of seconds simulated"),
    ADD_STAT(simTicks, statistics::units::Tick::get(),
             "Number of ticks simulated"),

• Group 實際上是一個樹狀的結構，以下面的例子，TlbStats 的 parent 會是 TLB。TLB 的 stats 就是 TLB::TlbStats

  // src/arch/arm/tlb.cc
  TLB::TlbStats::TlbStats(TLB &parent)
  : statistics::Group(&parent), tlb(parent),
  // src/arch/arm/tlb.cc
 TLB::TLB(const ArmTLBParams &p)
    : BaseTLB(p),
      table(name().c_str(), p.size, p.assoc,
            p.replacement_policy, p.indexing_policy),
      size(p.size),
      isStage2(p.is_stage2),
      _walkCache(false),
      tableWalker(nullptr),
      stats(*this)

• parent-Group 可以選擇使用 addStatGroup 來新增一組 Group，或是 mergeStatGroup 把 Group 給 merge 起來。
• LifeCycle
  • derived class ( SimObject, or custom stat structs ) 可以 override 下面這些 virtual methods，以便管理自己底下的 statistics。
  • regStats()
    • 在 simulator 初始化的時候會被呼叫
  • resetStats()
    • 當我們想要 reset statistics 的時候呼叫 ( e.g., at the end of a checkpoint restore or when resetting stats via a simulator command )
  • preDumpStats()
    • 在把 statistics 輸出到 stats.txt 之前會呼叫。
• 與 SimObject 的整合
  • 因為 class SimObject 也會繼承 statistics::Group。這代表每一個 simulation object 自動也會變成是一個 statistics Group。
  • 最後 statistics 可以根據 python-configuration-script ( e.g. configs/example/arm/starter_fs.py ) 的 topology 自動呈現出正確的 hierarchy。
  • 以 system.cpu_cluster.cpus0.ipc 為例
    • system, cpu_cluster, cpus0 是 Group
    • ipc 是一個 stat

class Drainable

{ https://github.com/gem5/gem5/blob/stable/src/sim/drain.hh }

• 這個 class 跟 Drain 這個操作有關，可以等之後需要使用 Drain 的時候，再來仔細研究這個 class
• 什麼是 Drain
  • In gem5, draining is the process of bringing the simulator into a consistent, "quiet" state where all in-flight operations (such as memory requests, cache transitions, or CPU pipeline stages) have completed, and no new events are being generated.
  • 當模擬器因為某些原因，可能需要停下來的時候，需要抵達一個的穩定狀態，這時候需要把還沒完成的請求 ( e.g. memory requests, cache transitions, CPU pipeline stages… ) 執行完成，並且在執行這些請求的時候，不接收任何新的請求。
  • Simulation in gem5 is event-driven. At any point in time, there might be many events scheduled in the future and data packet transmissions in-flight. To safely save the state of the simulator (checkpointing) or change the configuration mid-run, the simulator must reach a "quiet" or consistent state where no transactions are in-flight. This process is called draining.
• 什麼時候需要 Drain
  • 例如我們想要做一個 checkpoint，將 gem5 的執行狀態儲存到 host mahcine 的 disk。為了讓我們在 restore 的時候，可以正確的 restore，整個系統必須要在一個 乾淨 (clean) 的狀態，在這個乾淨的狀態裡面，不可以有 in-flight events ( e.g. 一個 packet 仍然在 bus 中傳遞 )。
  • 我們想要進行 CPU Switching ( Fast-Forwarding ) 的時候，也會需要透過 Drain 進到 乾淨 的狀態。
    • Fast-Forwarding : 先用不精準但快速的 CPU model 跑到感興趣的部分( Region of Interest, ROI )，然後再使用精準但很慢的 CPU model 來取得 ROI 的效能數字。
    • 例如，通常我們對 booting OS 的過程中的效能數字不感興趣，感興趣的通常會是 benchmark ( e.g. SPEC 2006 )，這時候就可以用快速的 CPU model ( AtomicSimpleCPU ) 完成 linux kernel 的 booting，然後再切換到慢速的 CPU model ( e.g. minor, O3CPU ) 測量效能數據。
• Key Components
  • DrainState ( Enum )
    • Represents the current state of a Drainable object
    • Running : 這個 object 正常的運行著
    • Draining : 這個 object 被要求開始 Drain 了，但是仍需要更多的 simulation time 來讓內部的狀態抵達 乾淨 的狀態 ( e.g., waiting for an in-flight memory request to return )。
    • Drained : 這個 object 已經到了 乾淨 ( clean ) 狀態，可以開始準備執行 simulator event ( e.g. checkpoint )
    • Resuming : 當 simulation 準備轉換回 Running 狀態的短暫狀態 ( transient state )。
  • Drainable ( Interface Class )
    • Any class that needs to handle draining must inherit from Drainable (note that SimObject inherits from Drainable , so most simulation objects have this capability).
    • virtual DrainState drain() = 0 : Derived classes must implement this. When called, the object should start flushing its state. If it can do so instantly, it returns DrainState::Drained . If it needs more time, it returns DrainState::Draining and must later signal completion.
    • virtual void drainResume() : Called when simulation is resuming. Derived classes can override this to restart their normal operations (e.g., rescheduling events).
    • void signalDrainDone() : Called by the object itself when it was in the Draining state and has finally finished clearing its state. This notifies the DrainManager .
    • virtual void notifyFork() : A hook called in the child process after a process fork, allowing the object to handle shared resources (like reopening files).
  • DrainManager ( Class )
    • A singleton class ( DrainManager::instance(), 在 gem5 只會有一個 instance ) that coordinates the draining process across the entire simulator.
    • It maintains a list of all instantiated Drainable objects.
    • tryDrain() : Initiates the drain process by calling drain() on all registered objects. It returns true if all objects are immediately Drained , or false if some enters the Draining state.
    • If tryDrain() returns false , the simulation loop continues until all objects that returned Draining have called signalDrainDone().
• Drain 大致上的過程
  • The simulation script requests a drain (e.g., before checkpointing)
  • DrainManager 會協調各個 Drainable object ( e.g. SimObject ) 。DrainManager 的 DrainManager::tryDrain() 會呼叫系統內的每一個 Drainable object 的 drain function。( e.g. src/cpu/minor/cpu.cc/MinorCPU::drain )
    • 假如 SimObject 全部都進入 Drained status，可以 return true，表示沒有任何 pending work 了
    • 否則 return false，表示仍有 in-flight operations。SimObject 進入 Draning status 後，通常會禁止所有新的 input，並且處理 pending requests
  • 假如還有 SimObject 仍不是 Drained status，則 simulation loop 會繼續下去。
  • 所有 SimObject 都進入 Drained status 之後，會停止 simulation loop。
  • simulator 可以開始進行 checkpoint 或是 CPU-switch
  • DrainManager::resume() 被呼叫的時候，會嘗試讓每一個 Drainable 呼叫 drainResume()，嘗試恢復 simulation 的執行。

class Serializable

{ https://github.com/gem5/gem5/blob/stable/src/sim/serialize.hh }

• 這個 class 跟 checkpoint 有關，可以等之後需要使用 checkpoint 的時候，再來仔細研究這個 class
• What does this class do ?
  • Serializable interface 是用來創建 checkpoints。任何實作這個 interface 的物件，都可以被列入 gem5 的 checkpoint system 裡面。
  • The serialize.hh class in gem5 is the base interface for any object that needs to save its state to a checkpoint (serialization) and restore its state from a checkpoint (unserialization)
• Methods
  • 為了讓一個 class 是可以被序列化的 ( serializable )，它必須要繼承 Serializable，並且實作兩個 pure virtual methods.
  • virtual void serialize(CheckpointOut &cp) const = 0
    • This method defines how the object writes its internal state to the checkpoint. CheckpointOut is a type alias for std::ostream .
  • virtual void unserialize(CheckpointIn &cp) = 0
    • This method defines how the object restores its internal state from the checkpoint. CheckpointIn wraps an IniFile parser to read key-value pairs from the checkpoint file.
• Helper Macros
  • To reduce boilerplate code in serialize and unserialize implementations, serialize.hh defines a set of helper macros.
  • Scalars
    • SERIALIZE_SCALAR(var) / UNSERIALIZE_SCALAR(var)
  • Enums
    • SERIALIZE_ENUM(var) / UNSERIALIZE_ENUM(var) (casts to int on output)
  • Arrays
    • SERIALIZE_ARRAY(arr, size) / UNSERIALIZE_ARRAY(arr, size)
  • Containers (vector, set, etc.)
    • SERIALIZE_CONTAINER(container) / UNSERIALIZE_CONTAINER(container)
  • Sub-objects
    • SERIALIZE_OBJ(obj) / UNSERIALIZE_OBJ(obj) (calls serializeSection / unserializeSection on the sub-object)

class EventQueue

{ https://github.com/gem5/gem5/blob/stable/src/sim/eventq.hh }

• 這邊先不去探討 multi-thread gem5 simulation 這類進階的議題。
  • multi-thread gem5 simulation 可以讓每個 thread 有自己的 eventq
  • 其實內涵跟 SystemC 的 loosely-timed simulation 有點像，在 SystemC 的 loosely-timed simulation，一個 SC_thread 所推進的時間，可以比 global 的時間還要多 ( e.g. loosely timed 的 CPU model 可以先擅自推進 5 毫秒，而 global 的時間可能仍在 0。 )，然後每隔一個 Quantum 的時間，再去 sync 每一個 SC_thread 的時間。SystemC 一個可能的實作是，每次一個 loosely-timed 的 SC_thread 需要推進時間的時候，只把時間記錄在自己的 thread 裡面的某個變數，等到這個變數的值達到 Quantum 之後，再一口氣 wait 一個 Quantum 的時間。
• 這個 class 在做什麼 ?
  • The EventQueue class, defined in eventq.hh, is the core engine of gem5's discrete-event simulation. It maintains a list of simulation events sorted by their scheduled trigger time (Tick) and priority.
• Synchronous v.s. Asynchronous / Scheduling
  • Synchronous Events
    • 目前重點會需要先學習 Synchronous Events
    • Scheduled by the thread that owns the EventQueue . These are inserted directly into the main event queue ( head ) and are deterministic.
  • Asynchronous Events
    • Scheduled by other threads (or marked as global ). To avoid race conditions and maintain determinism, these are initially placed in a thread-safe async_queue and merged into the main queue at the end of each simulation quantum (synchronization step) via handleAsyncInsertions()
• Member Variables
  • head
    • Pointer to the first Event in the queue (the event scheduled for the earliest tick),
    • 假如 tick 相同 ( 兩個時間插入的時間相同 )，則 priority 的值越小，優先度越高。
  • _curTick
    • The current simulation time (in Ticks) for this queue.
  • async_queue
    • A std::list storing events inserted by other threads.
  • async_queue_mutex
    • Protects async_queue from concurrent access.
  • service_mutex
    • Lock that protects the main queue during event servicing.
• Methods
  • schedule(Event *event, Tick when, bool global=false)
    • Schedules an event to run at when. If called from a different thread or if global is true, it inserts it into the async_queue. Otherwise, it inserts it directly into the main queue.
  • deschedule(Event *event)
    • Removes a scheduled event from the queue. Must be called by the owning thread.
  • reschedule(Event *event, Tick when, bool always=false)
    • Changes the scheduled time of an event. Must be called by the owning thread.
  • serviceOne()
    • Processes the event at the head of the queue. It updates _curTick to the event's tick and calls event->process() .
  • serviceEvents(Tick when)
    • Services all events up to the specified when tick.
  • handleAsyncInsertions()
    • Moves events from the async_queue to the main queue. This is typically called at synchronization points.
• 從 src/sim/eventq.cc 可以看到 std::vector<EventQueue *> mainEventQueue;，可以得知 mainEventQueue 實際上有很多個。
  • 而根據 src/sim/Root.py/class Root，eventq_index 預設為 0
  • 並且 src/python/m5/SimObject.py/class SimObject 的 eventq_index = Param.UInt32(Parent.eventq_index, "Event Queue Index")，可以知道，SimObject 的 eventq_index 會從 parent object 拿。
  • 於是大部份的 SimObject 所使用的 eventq_index 也是 0。

class EventManager

{ https://github.com/gem5/gem5/blob/stable/src/sim/eventq.hh }

• 這個 class 是做什麼 ?
  • 在 gem5 的 descrete event simulation 中，很多 objects ( 大部分繼承自 SimObject ) 需要呼叫 schedule, deschedule 或是 reschedule 來調整 events 在 event-queue 中的狀態。
  • gem5 的設計不讓 SimObject 自己管理 指向 EventQueue 的指標，並呼叫 EventQueue 的 method。取而代之地，SimObject 繼承 EventManager。
  • EventManager 可以幫助 SimObject 把對 EventQueue 的操作 ( e.g. scheduler, deschedule, reschedule… )，委託給正確的 EventQueue。
  • In gem5, simulation time progresses by executing events scheduled on event queues. In multi-core or multi-threaded simulations, there can be multiple event queues running in parallel (typically one per simulation thread).
  • Instead of forcing simulation components to manually keep track of which queue they should use and access it directly, components inherit from EventManager .
  • SimObject Integration
    • The base class for most gem5 components, SimObject , inherits from EventManager.
    • During configuration and instantiation, each SimObject is assigned to a specific EventQueue (stored in EventManager::eventq ).
  • Convenience
    • By inheriting from EventManager , any SimObject can schedule its own events simply by calling: schedule(myEvent, curTick() + delay)
  • This automatically routes the event to the correct queue without the object needing to know which queue it is on.
• Member
  • eventq : A protected pointer to the EventQueue associated with this manager.
• Constructors
  • EventManager(EventQueue *eq)
    • Associates the manager with a specific event queue.
  • EventManager(EventManager &em) / EventManager(EventManager *em)
    • Copies the event queue pointer from another manager.
• Public Methods
  • eventQueue()
    • Returns the underlying EventQueue pointer.
  • schedule(Event &event, Tick when) / schedule(Event *event, Tick when)
    • 讓一個 event 會在一個特定的 simulation tick 被觸發
  • deschedule(Event &event) / deschedule(Event *event)
    • 把一個 event 從 event queue 上移除
  • reschedule(Event &event, Tick when, bool always = false) / reschedule(Event *event, Tick when, bool always = false)
    • 更改一個 event 觸發的時間。
  • setCurTick(Tick newVal)
    • 把某一個 Event Queue 的時間設定到某個特定的 simulation tick。
    • Sets the current tick of the associated event queue.
  • EventManger provides overload methods for schedule, deschedule, and reschedule that take either references or pointers to Event objects and forward the calls to the underlying event queue.
• Delegation
  • When you call a method like schedule() on an EventManager instance, it simply forwards the call to the underlying EventQueue pointed to by eventq
• Inheritance
  • This class is designed to be inherited. For example, SimObject (the base class for almost all simulation objects in gem5) inherits from EventManager, allowing any simulation object to easily schedule its own events (e.g., schedule(tickEvent, nextCycle) ).

class SimObject

{ https://github.com/gem5/gem5/blob/stable/src/sim/sim_object.hh#L146 }
{ https://github.com/gem5/gem5/blob/stable/src/python/m5/SimObject.py }

• 這個 class 在做什麼 ?
  • SimObject 是一個基礎的 abstract base class，幾乎所有 simulation components ( e.g. CPUs, caches, buses… ) 都會繼承 SimObject。
  • 只要繼承了 SimObject，我們便能在 python configuration 裡面，去使用這個 simulation components，並可以對 simulation components 進行設定。
• Inheritance
  • SimObject integrates several core gem5 systems by inheriting from their respective base classes。這部分可以看 sim_object.hh
  • class SimObject : public EventManager, public Serializable, public Drainable, public statistics::Group, public Named
  • Named
    • Gives the object a name string (accessible via name() ), which is crucial for identification and debug output formatting.
  • EventManager
    • Associates the object with an EventQueue and allows it to schedule simulation events (e.g., schedule(tickEvent, curTick() + 1) ).
  • statistics::Group
    • Makes the object a node in the hierarchical statistics tree, allowing it to register and organize its own statistics (e.g., cache hits/misses).
  • Serializable
    • Provides methods ( serialize / unserialize ) to save the object's state to a checkpoint and restore it later.
  • Drainable
    • Provides methods ( drain / drainResume ) to pause the object and flush its in-flight transactions before checkpointing or CPU handovers.
• Python-C++ Binding and Parameters
  • gem5 的模擬，是在 python 進行 configuration，但卻是在 C++ 進行模擬。而 SimObject 則扮演了 Python 與 C++ 間的橋樑。
  • Params
    • For every SimObject class (e.g., Foo ), gem5 auto-generates a parameter struct (e.g., FooParams ) containing configuration values defined in Python.
    • 以 src/cpu/minor/BaseMinorCPU.py/BaseMinorCPU，gem5 會自動產生一個 build/ARM/params/BaseMinorCPU.hh/BaseMinorCPUParams。當我們在 python configuration script 裡面去設定 threadPolicy 這個參數的時候，我們可以在 C++ 這一側透過 params.threadPolicy 去拿到該設定的值。
  • Constructor
    • The C++ constructor of a SimObject always takes a reference to its parameter struct: SimObject(const Params &p) .
  • PARAMS Macro ( declare in src/sim/sim_object.hh )
    • Derived classes use the PARAMS(ClassName) macro to easily define a params() method that returns their specific, downcasted parameter struct.
    • 雖然好像蠻少人在用的 …
• Lifecycle
  • During simulator startup, objects go through a strict initialization sequence.
  • Instantiation
    • C++ objects are created using the create() method in their generated Params struct ( e.g. build/ARM/params/BaseAtomicSimpleCPU.hh ).
  • Port Connection
    • Inter-object ports are connected.
  • init()
    • Virtual method called after all objects are created and connected. Used for basic C++ initializations.
  • regStats()
    • Called to register statistics.
  • State Restore
    • initState()
      • Called for a "cold start" (no checkpoint).
    • loadState(cp)
      • Called when restoring from a checkpoint.
  • resetStats()
    • Resets statistics before the run.
  • startup()
    • Called just before simulation begins. This is where objects typically schedule their initial events (like the first CPU tick).
    • e.g. src/cpu/minor/cpu.cc/MinorCPU::startup
• Methods
  • init() / startup()
    • For custom initialization.
  • getPort(name, index)
    • Must be overridden if the object has ports to connect to other objects. It returns a reference to the requested port.
  • serialize() / unserialize()
    • To support checkpointing.
  • drain()
    • To support pausing the simulation safely.
  • memWriteback() / memInvalidate()
    • Used by memory-related objects (like caches) to flush dirty data to memory or invalidate cache lines before checkpoints or CPU model switches.
• Debugging Support
  • simObjectList
    • A static list of all instantiated SimObjects in the simulation.
  • static SimObject *find(const char *name)
    • A static helper function that allows developers to find a pointer to a specific C++ object by its full hierarchical name, which is highly useful when debugging with gdb .
    • 這邊簡單來 demo 一下如何在 gdb 裡面去使用 find。這樣子我們可以透過 name 來取得任意 SimObject 的指標了。想知道有哪些 name，可以看 stats.txt

# 首先要先 backtrace，看一下哪一層會在 SimObject 的範圍
(gdb) bt
# 這邊假設在 frame 2 的位置，以我的情況，會在 `gem5::AtomicSimpleCPU` 這個 `SimObject` 的範圍
(gdb) frame 2
(gdb) p this
$1 = (gem5::AtomicSimpleCPU * const) 0x555565050e00
(gdb) p (SimObject*)this
$2 = (gem5::SimObject *) 0x555565050e00
(gdb) p (SimObject*)this->find
$3 = (gem5::SimObject *) 0xec814853e5894855
(gdb) p this->name()
$8 = "system.cpu_cluster.cpus"
(gdb) p (SimObject*)this->find("system.cpu_cluster.cpus")
$9 = (gem5::SimObject *) 0x555565050e00
(gdb) p (SimObject*)this->find("system.mem_ctrls.dram")
$2 = (gem5::SimObject *) 0x55556535a900
(gdb) p (SimObject*)this->find("system.mem_ctrls")
$3 = (gem5::SimObject *) 0x55556dff2000
(gdb) p (SimObject*)this->find("system")
$4 = (gem5::SimObject *) 0x55556482f700