Transcript
A Fully Buffered Memory System Simulator
Rami Nasr -M.S. Thesis, and ENEE 759H Course Project Thursday May 12th, 2005
Another Simulator? Sim-DRAM exists and supports FB-DIMM. Why write another simulator?
Sim-DRAM still had a few unworkable bugs in its FB-DIMM model when I began my study. FB-DIMM is radically different than other memory architectures. New simulator => fresh start. FBsim is made exclusively for simulating and studying the FBDIMM architecture. Easier to study FB-DIMM with an exclusive simulator. Different scheduler, mapping algorithm, approach, style, section of study in the FB-DIMM design space. FBsim is ideal for simulating ‘unreasonably’ high memory request rates and studying channel saturation effects. The two simulators can be used to validate each other’s results in FB-DIMM studies. Writing a memory simulator was a great experience for me.
FBsim Overview
All code written from scratch. Standalone product. Does not currently interface with CPU simulators or memory traces. Instead probabilistically models memory transactions according to user specifications.
=> Does not actually store memory data
Written in ANSI C. ~5000 lines of code. Code organized into header files, commented, quite easy to hack.
Fast. For each memory channel, 1 second simulates ~10ms (or ~1ms during channel saturation) on a 2.4 GHz Pentium 4.
Supports Open & Closed Page Mode, Fixed & Variable Latency Mode.
Supports output of macro and micro (frame by frame) simulation data
Does not model channel init, maintenance, sync. overhead.
Does not model memory refresh.
Does not model power consumption, and power timing limitations (tFAW etc.).
The above options can be incorporated readily into future versions.
FBsim Overview 2 Channel Scheduler 0
Input Transaction Generator
Channel Scheduler 1 Address Mapper
A Frame Iteration •Try to generate transactions •Map any generated transactions to its channel scheduler. •Fire each scheduler once.
Channel Scheduler 7
Input Transaction Model • Step Distributions
• Normal
(Gaussian) Distributions
Input Transaction Model 2 Bus Trace Viewer
FBsim Model
Address Mapping
Physical addressClosed mustWHILE be mapped somehow to the Page (a Mode non zero row sum exists) right channel, DIMM, rank, bank, row, and column. {
WHILE (visit each channel with a non zero row sum exactly once) FBsim built to support different DIMM capacities, { Theeven next 'result' is channel DIMM with the different channel capacities, unbalanced highest number. configurations Decrement that DIMM's number by 1. Decrement the row sum by 1. Mode } Modulus = 4+2+1+2 = 9Open Page => Algorithm needed }to map incoming transaction to
DIMM
Channel Scheduler
FB-DIMM Frame Format Review
SouthBound (SB) Frame could be a: • Channel Frame (not modeled in FBsim) • Command Frame (up to three DRAM commands, with only one command possible to each DIMM in the channel) • Command + Wdata Frame (holds one DRAM command, plus one DDR beat of write data)
NorthBound (NB) Frame could be a: • Channel Frame (not modeled in FBsim)
• Read Response Frame (holds two DDR beats of returned read data)
Some of my Results Case Study Conclusion • With at least two DIMMs on each channel, performance scales very well in FB-DIMM •More than two DIMMs only increases throughput
• 1x8 achieved 7.9 GBps before saturating (82%)
• 2x4 achieved capacity, 15.6not GBps (82%)
• 4x2 achieved •Adding each DIMM adds ~5ns average channel latency 31.3 GBps in FLM, and slightly over half that in VLM (82%) • 8x1 achieved • In closed page mode, only 82% of peak theoretical 45.2 GBps throughput of a channel can be reached. (59%!)
Some of my Results 2 • In Closed Page Mode with 2:1 read/write ratio, a reordering window of size ~12 transactions achieves best possible performance (channel saturation) for a FB-DIMM channel scheduler. Increasing window-size over this has no benefit. • The more skewed the read/write ratio, the bigger the scheduling window needs to be (at 4:1, its ~18). • In Variable Latency Mode, a reordering window of size ~20 achieves best possible performance.
Some of my Results 3 Micro-study shows that in Closed Page Mode, the FB channel can at most reach ~93% write data utilization on the SB, and ~84% read data utilization on the NB. Micro-study showed that FBsim channel utilization was slightly worse for non 2:1 read/write ratios (it was 2% worse for 4:1). FBsim scheduler can quite straightforwardly be made more adaptive to read/write ratio of transactions in scheduler.
Future Ideas with FBsim • I’m graduating this semester (if Dr Jacob and Mr (Dr?) Wang so please), and escaping to the corporate world. • => Writing a guide for FBsim along with some ideas for future work. Anyone who wishes to take over development is eagerly encouraged to. • If so, I would be happy to help get things rolling by email or in person. Feel free to access & use anything in FBsim or my thesis paper. • I strongly believe a very interesting paper or three can quite quickly come out of this research area
(me)
Future Ideas with FBsim 2 • For credibility in a paper, add an interface between FBsim and a CPU simulator or memory traces. Run real benchmarks through FBsim. Compare and contrast these results with the transaction modeling results. • AND/OR add more functionality and provable realism to the transaction modeler. Study this. • Best yet, integrate FBsim into the Sim-DRAM package as an added option. • Add modeling for channel overhead, memory refresh overhead, error simulation and error handling, power consumption constraints and metrics. • Enhance adaptivity of FBsim scheduler to non 2:1 read/write ratios. • Experiment with address mapping algorithm and load balancing. • Experiment with different type scheduler implementations (eg. ones not based on pattern matching). *involved* • Study hardware constraints in FB-DIMM channel scheduling.
More Possible FB-DIMM Studies
Channel utilization and configuration trade-offs for Open Page Mode Performance degradation of shrinking scheduler reorder window size Relaxation on critical DRAM device parameters (density, nBanks, timing constraints, clock frequency) allowed by FB-DIMM architecture OR optimizing the FB-DIMM architecture by increasing the SB and NB channel widths (adding lines) or bitrates, and maybe modifying the frame protocol AMB is a logic device on a memory module!! Can add buffers, arithmetic units, processing power, etc…..
Special Thanks to..
Dr Jacob for introducing me to the field and guiding my progress David Wang for the course lectures and material
