On a busy OLTP system, the database often spends more time waiting for writes than you would expect from a machine full of flash. One reason is that Exadata's flash cache can run in two modes, and in one of them database writes still go to the hard disks.
This post explains write-through and write-back flash cache: where a write actually goes in each mode, what write-back really speeds up (and the common misconception about commits), how data stays safe, and how to check, enable, revert and monitor it.
What is Exadata Smart Flash Cache?
Every Exadata storage server ("cell") contains fast flash devices (NVMe on current models) in front of its hard disks. Exadata Smart Flash Cache uses that flash as a cache, and it is smart in the literal sense: each database I/O carries a hint about how likely the data is to be reused. The cell uses that hint to decide what to keep in flash. A full backup or a one-time table scan, for example, is not allowed to flush your hot OLTP blocks out of the cache.
The two flash cache modes
Write-Through
Database writes go to the hard disks, and the write is acknowledged only once the disk has it. Flash is used to accelerate reads. This is the default value of the cell attribute flashCacheMode.
Write-Back
Database writes go to flash first and are acknowledged as soon as they are safely on flash, mirrored by ASM to other cells just like data on disk. The cell copies ("destages") the data to hard disk later, in the background. Hot blocks that are written again and again may stay in flash for a long time, and every overwrite is absorbed by flash instead of costing a disk write.
Write-Through vs Write-Back at a glance
| Write-Through | Write-Back | |
|---|---|---|
| Where writes land first | Hard disk | Flash (mirrored across cells) |
| Write latency | Disk latency | Flash latency, much lower |
| Repeated writes to a hot block | Each one goes to disk | Absorbed in flash; disk sees far fewer writes |
| Reads | Served from flash when cached | Served from flash when cached |
| After a cell restart | Cache is rebuilt (warm-up) | Cache content persists, no warm-up |
| Protection | Data is on disk (ASM mirrored) | Data on flash is ASM-mirrored; Oracle recommends HIGH redundancy |
| Typical fit | Read-heavy, DW/analytics, low write pressure | Write-heavy OLTP, high DBWR load, "free buffer waits" |
What Write-Back really improves (and what it does not)
Redo: Smart Flash Log and Smart Flash Log Write-Back
| Feature | What it does | Needs |
|---|---|---|
| Smart Flash Log | Writes redo to flash and disk at the same time and acknowledges whichever finishes first, removing log-write latency outliers | On by default |
| Smart Flash Log Write-Back | Keeps the whole contents of the redo logs in write-back flash cache, so HDD bandwidth no longer limits redo throughput. Oracle quotes up to 250% higher log write throughput, depending on workload | Exadata System Software 20.1+, X7 or later, write-back flash cache, database in ARCHIVELOG mode |
So write-back flash cache matters for redo too, but indirectly: it is the prerequisite for Smart Flash Log Write-Back, which helps systems where heavy redo generation, archiving, GoldenGate or RMAN saturate the hard disks.
Is it safe?
The usual worry is: "if the write only reached flash, what happens when a flash card fails?" Exadata handles it in three ways:
- Mirroring: writes to write-back cache are protected by ASM mirroring across storage servers, exactly like data on disk. If a flash device fails, reads transparently switch to the mirror copy and the lost data is rebuilt from it. No user action is needed.
- Persistence: write-back cache content survives a cell restart; it is flash, not RAM.
- Redundancy: Oracle recommends HIGH redundancy (three copies) for databases using write-back flash cache. Oracle's deployment tool (OEDA) has used write-back as its default for new High Capacity deployments with HIGH redundancy since 2017. On Extreme Flash storage, write-back is always on and cannot be disabled.
When to use Write-Back
| Good fit | Probably no gain |
|---|---|
| Write-intensive OLTP with high DBWR activity | Read-mostly or analytic workloads |
| Significant free buffer waits, high db file parallel write times | Systems with low write I/O and healthy write latency |
| Hot tables or indexes updated many times | Bulk loads that write each block once (direct path) |
| Heavy redo throughput (to enable Smart Flash Log Write-Back) | — |
Check the current mode
# On one storage cell
CellCLI> LIST CELL ATTRIBUTES flashCacheMode
WriteBack
# On all cells at once, from a database server
dcli -g cell_group -l root "cellcli -e list cell attributes flashCacheMode"
# Which grid disks are cached by which flash
dcli -g cell_group -l root "cellcli -e list griddisk attributes name,cachingPolicy,cachedBy"Enable Write-Back (Exadata 11.2.3.3.1 and later)
From release 11.2.3.3.1, you no longer need to stop cell services or inactivate grid disks to switch to write-back. You do need to drop and recreate the flash cache, and the documented sequence is:
# 1. All physical disks must be NORMAL (this should return nothing) dcli -g cell_group -l root "cellcli -e list physicaldisk attributes name,status" | grep -v NORMAL # 2. Drop the flash cache dcli -g cell_group -l root "cellcli -e drop flashcache" # 3. Switch the mode dcli -g cell_group -l root "cellcli -e alter cell flashCacheMode=writeback" # 4. Recreate the flash cache dcli -g cell_group -l root "cellcli -e create flashcache all" # 5. Verify dcli -g cell_group -l root "cellcli -e list cell detail" | grep flashCacheMode
Go back to Write-Through
The reverse needs one extra step: dirty data in flash must first be flushed to disk. The cell refuses to switch to write-through while a flash cache with unflushed data exists.
CellCLI> ALTER FLASHCACHE ALL FLUSH CellCLI> LIST GRIDDISK ATTRIBUTES name, flushstatus, flusherror -- wait until every disk shows completed CellCLI> DROP FLASHCACHE CellCLI> ALTER CELL flashCacheMode=writethrough CellCLI> CREATE FLASHCACHE ALL
Flushing can take a long time on a busy system. Follow the procedure in the Exadata Database Machine Maintenance Guide for your release, including any rolling steps, before doing this in production.
Monitor write-back activity
On the storage cells
CellCLI> LIST METRICCURRENT WHERE objectType = 'FLASHCACHE' -
AND name LIKE 'FC_(BY_DIRTY|IO_BY_W|IO_RQ_W|IO_RQ_W_OVERWRITE|IO_BY_DISK_WRITE)'
| Metric | Meaning |
|---|---|
FC_BY_DIRTY | MB in flash not yet written to disk. Normal on write-back; watch for steady growth |
FC_BY_STALE_DIRTY | Dirty MB that cannot be flushed because the disk is not reachable. Should be 0 |
FC_IO_BY_W / FC_IO_RQ_W | Data and requests written into flash cache |
FC_IO_RQ_W_OVERWRITE | Overwrites absorbed by flash: the direct benefit of write-back |
FC_IO_BY_DISK_WRITE | Data destaged from flash to hard disk |
FC_IO_BY_W_SKIP | Writes that bypassed flash and went straight to disk |
In the database
SELECT name, value
FROM v$sysstat
WHERE name IN ('cell writes to flash cache',
'cell flash cache read hits',
'physical write total IO requests');
SELECT event, total_waits, ROUND(time_waited_micro/NULLIF(total_waits,0)) AS avg_us
FROM v$system_event
WHERE event IN ('free buffer waits','db file parallel write','log file sync','log file parallel write');
AWR reports on Exadata also include a dedicated Exadata statistics section with flash cache hit ratios and per-cell I/O. Compare a snapshot before and after the change.
Quick FAQ
Q. Does write-back flash cache need a licence?
It is a feature of Exadata System Software, which is licensed with the storage servers. No extra database option is needed.
Q. Do I need to change anything in the database?
No. The change is entirely on the storage cells and transparent to the database.
Q. Does it help direct-path loads?
Usually not much: large sequential writes that touch each block once gain little from caching.
Q. What about Exadata Cloud / Exadata Database Service?
Oracle manages the storage tier there; check the service documentation for the configured mode instead of changing it yourself.
Final thoughts
Write-back flash cache turns Exadata's flash from a read cache into a write cache as well. For write-heavy OLTP it is often one of the simplest big wins available: DBWR writes complete at flash speed, overwrites never reach the disks, and the disks are freed for everything else. Protect it with HIGH redundancy, measure free buffer waits and the FC metrics before and after, and keep in mind that commit latency belongs to redo and Smart Flash Log, not to the cache mode.
References
- Write-Back Flash Cache (Exadata System Software User's Guide)
- Enable Write Back Flash Cache for 11.2.3.3.1 or Higher (Maintenance Guide)
- Smart Flash Log Write-Back (What's New, 20.1)
- Monitoring Exadata Smart Flash Cache Using Exadata Metrics
- Write-Back Flash Cache with Exadata Smart Flash Cache
Tamilselvan Perumal · Oracle ACE Apprentice · Lead Product Engineer & Senior DBA
