2fa9e406ahrens * CDDL HEADER START
3fa9e406ahrens *
4fa9e406ahrens * The contents of this file are subject to the terms of the
5033f983ek * Common Development and Distribution License (the "License").
6033f983ek * You may not use this file except in compliance with the License.
7fa9e406ahrens *
8fa9e406ahrens * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
9fa9e406ahrens * or http://www.opensolaris.org/os/licensing.
10fa9e406ahrens * See the License for the specific language governing permissions
11fa9e406ahrens * and limitations under the License.
12fa9e406ahrens *
13fa9e406ahrens * When distributing Covered Code, include this CDDL HEADER in each
14fa9e406ahrens * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
15fa9e406ahrens * If applicable, add the following below this CDDL HEADER, with the
16fa9e406ahrens * fields enclosed by brackets "[]" replaced with your own identifying
17fa9e406ahrens * information: Portions Copyright [yyyy] [name of copyright owner]
18fa9e406ahrens *
19fa9e406ahrens * CDDL HEADER END
20fa9e406ahrens */
223f9d6adLin Ling * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
23af1d63aPaul Dagnelie * Copyright (c) 2019, Joyent, Inc.
24fa98e48Matthew Ahrens * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
2571cb1b7Saso Kiselkov * Copyright (c) 2014 by Saso Kiselkov. All rights reserved.
2601a059eRoman Strashkin * Copyright 2017 Nexenta Systems, Inc.  All rights reserved.
27fa9e406ahrens */
3044cb6abbmc * DVA-based Adjustable Replacement Cache
31fa9e406ahrens *
32ea8dc4beschrock * While much of the theory of operation used here is
33ea8dc4beschrock * based on the self-tuning, low overhead replacement cache
34fa9e406ahrens * presented by Megiddo and Modha at FAST 2003, there are some
35fa9e406ahrens * significant differences:
36fa9e406ahrens *
37fa9e406ahrens * 1. The Megiddo and Modha model assumes any page is evictable.
38fa9e406ahrens * Pages in its cache cannot be "locked" into memory.  This makes
39fa9e406ahrens * the eviction algorithm simple: evict the last page in the list.
40fa9e406ahrens * This also make the performance characteristics easy to reason
41fa9e406ahrens * about.  Our cache is not so simple.  At any given moment, some
42fa9e406ahrens * subset of the blocks in the cache are un-evictable because we
43fa9e406ahrens * have handed out a reference to them.  Blocks are only evictable
44fa9e406ahrens * when there are no external references active.  This makes
45fa9e406ahrens * eviction far more problematic:  we choose to evict the evictable
46fa9e406ahrens * blocks that are the "lowest" in the list.
47fa9e406ahrens *
48fa9e406ahrens * There are times when it is not possible to evict the requested
49fa9e406ahrens * space.  In these circumstances we are unable to adjust the cache
50fa9e406ahrens * size.  To prevent the cache growing unbounded at these times we
51fa94a07brendan * implement a "cache throttle" that slows the flow of new data
52fa94a07brendan * into the cache until we can make space available.
53fa9e406ahrens *
54fa9e406ahrens * 2. The Megiddo and Modha model assumes a fixed cache size.
55fa9e406ahrens * Pages are evicted when the cache is full and there is a cache
56fa9e406ahrens * miss.  Our model has a variable sized cache.  It grows with
57fa94a07brendan * high use, but also tries to react to memory pressure from the
58fa9e406ahrens * operating system: decreasing its size when system memory is
59fa9e406ahrens * tight.
60fa9e406ahrens *
61fa9e406ahrens * 3. The Megiddo and Modha model assumes a fixed page size. All
62f717074Will Andrews * elements of the cache are therefore exactly the same size.  So
63fa9e406ahrens * when adjusting the cache size following a cache miss, its simply
64fa9e406ahrens * a matter of choosing a single page to evict.  In our model, we
65fa9e406ahrens * have variable sized cache blocks (rangeing from 512 bytes to
66f717074Will Andrews * 128K bytes).  We therefore choose a set of blocks to evict to make
67fa9e406ahrens * space for a cache miss that approximates as closely as possible
68fa9e406ahrens * the space used by the new block.
69fa9e406ahrens *
70fa9e406ahrens * See also:  "ARC: A Self-Tuning, Low Overhead Replacement Cache"
71fa9e406ahrens * by N. Megiddo & D. Modha, FAST 2003
72fa9e406ahrens */
75fa9e406ahrens * The locking model:
76fa9e406ahrens *
77fa9e406ahrens * A new reference to a cache buffer can be obtained in two
78fa9e406ahrens * ways: 1) via a hash table lookup using the DVA as a key,
79fa94a07brendan * or 2) via one of the ARC lists.  The arc_read() interface
805602294Dan Kimmel * uses method 1, while the internal ARC algorithms for
81f717074Will Andrews * adjusting the cache use method 2.  We therefore provide two
82fa9e406ahrens * types of locks: 1) the hash table lock array, and 2) the
835602294Dan Kimmel * ARC list locks.
84fa9e406ahrens *
85fc98feaBart Coddens * Buffers do not have their own mutexes, rather they rely on the
86fc98feaBart Coddens * hash table mutexes for the bulk of their protection (i.e. most
87fc98feaBart Coddens * fields in the arc_buf_hdr_t are protected by these mutexes).
88fa9e406ahrens *
89fa9e406ahrens * buf_hash_find() returns the appropriate mutex (held) when it
90fa9e406ahrens * locates the requested buffer in the hash table.  It returns
91fa9e406ahrens * NULL for the mutex if the buffer was not in the table.
92fa9e406ahrens *
93fa9e406ahrens * buf_hash_remove() expects the appropriate hash mutex to be
94fa9e406ahrens * already held before it is invoked.
95fa9e406ahrens *
965602294Dan Kimmel * Each ARC state also has a mutex which is used to protect the
97fa9e406ahrens * buffer list associated with the state.  When attempting to
985602294Dan Kimmel * obtain a hash table lock while holding an ARC list lock you
99fa9e406ahrens * must use: mutex_tryenter() to avoid deadlock.  Also note that
10044eda4dmaybee * the active state mutex must be held before the ghost state mutex.
101fa9e406ahrens *
102fa9e406ahrens * Note that the majority of the performance stats are manipulated
103fa9e406ahrens * with atomic operations.
104fa94a07brendan *
10589c86e3Chris Williamson * The L2ARC uses the l2ad_mtx on each vdev for the following:
106fa94a07brendan *
107fa94a07brendan *	- L2ARC buflist creation
108fa94a07brendan *	- L2ARC buflist eviction
109fa94a07brendan *	- L2ARC write completion, which walks L2ARC buflists
110fa94a07brendan *	- ARC header destruction, as it removes from L2ARC buflists
111fa94a07brendan *	- ARC header release, as it removes from L2ARC buflists
112fa9e406ahrens */
114dcbf3bdGeorge Wilson/*
115dcbf3bdGeorge Wilson * ARC operation:
116dcbf3bdGeorge Wilson *
117dcbf3bdGeorge Wilson * Every block that is in the ARC is tracked by an arc_buf_hdr_t structure.
118dcbf3bdGeorge Wilson * This structure can point either to a block that is still in the cache or to
119dcbf3bdGeorge Wilson * one that is only accessible in an L2 ARC device, or it can provide
120dcbf3bdGeorge Wilson * information about a block that was recently evicted. If a block is
121dcbf3bdGeorge Wilson * only accessible in the L2ARC, then the arc_buf_hdr_t only has enough
122dcbf3bdGeorge Wilson * information to retrieve it from the L2ARC device. This information is
123dcbf3bdGeorge Wilson * stored in the l2arc_buf_hdr_t sub-structure of the arc_buf_hdr_t. A block
124dcbf3bdGeorge Wilson * that is in this state cannot access the data directly.
125dcbf3bdGeorge Wilson *
126dcbf3bdGeorge Wilson * Blocks that are actively being referenced or have not been evicted
127dcbf3bdGeorge Wilson * are cached in the L1ARC. The L1ARC (l1arc_buf_hdr_t) is a structure within
128dcbf3bdGeorge Wilson * the arc_buf_hdr_t that will point to the data block in memory. A block can
129dcbf3bdGeorge Wilson * only be read by a consumer if it has an l1arc_buf_hdr_t. The L1ARC
1305602294Dan Kimmel * caches data in two ways -- in a list of ARC buffers (arc_buf_t) and
131770499eDan Kimmel * also in the arc_buf_hdr_t's private physical data block pointer (b_pabd).
1325602294Dan Kimmel *
1335602294Dan Kimmel * The L1ARC's data pointer may or may not be uncompressed. The ARC has the
134770499eDan Kimmel * ability to store the physical data (b_pabd) associated with the DVA of the
135770499eDan Kimmel * arc_buf_hdr_t. Since the b_pabd is a copy of the on-disk physical block,
1365602294Dan Kimmel * it will match its on-disk compression characteristics. This behavior can be
1375602294Dan Kimmel * disabled by setting 'zfs_compressed_arc_enabled' to B_FALSE. When the
138770499eDan Kimmel * compressed ARC functionality is disabled, the b_pabd will point to an
1395602294Dan Kimmel * uncompressed version of the on-disk data.
1405602294Dan Kimmel *
1415602294Dan Kimmel * Data in the L1ARC is not accessed by consumers of the ARC directly. Each
1425602294Dan Kimmel * arc_buf_hdr_t can have multiple ARC buffers (arc_buf_t) which reference it.
1435602294Dan Kimmel * Each ARC buffer (arc_buf_t) is being actively accessed by a specific ARC
1445602294Dan Kimmel * consumer. The ARC will provide references to this data and will keep it
1455602294Dan Kimmel * cached until it is no longer in use. The ARC caches only the L1ARC's physical
1465602294Dan Kimmel * data block and will evict any arc_buf_t that is no longer referenced. The
1475602294Dan Kimmel * amount of memory consumed by the arc_buf_ts' data buffers can be seen via the
148dcbf3bdGeorge Wilson * "overhead_size" kstat.
149dcbf3bdGeorge Wilson *
1505602294Dan Kimmel * Depending on the consumer, an arc_buf_t can be requested in uncompressed or
1515602294Dan Kimmel * compressed form. The typical case is that consumers will want uncompressed
1525602294Dan Kimmel * data, and when that happens a new data buffer is allocated where the data is
1535602294Dan Kimmel * decompressed for them to use. Currently the only consumer who wants
1545602294Dan Kimmel * compressed arc_buf_t's is "zfs send", when it streams data exactly as it
1555602294Dan Kimmel * exists on disk. When this happens, the arc_buf_t's data buffer is shared
1565602294Dan Kimmel * with the arc_buf_hdr_t.
157dcbf3bdGeorge Wilson *
1585602294Dan Kimmel * Here is a diagram showing an arc_buf_hdr_t referenced by two arc_buf_t's. The
1595602294Dan Kimmel * first one is owned by a compressed send consumer (and therefore references
1605602294Dan Kimmel * the same compressed data buffer as the arc_buf_hdr_t) and the second could be
1615602294Dan Kimmel * used by any other consumer (and has its own uncompressed copy of the data
1625602294Dan Kimmel * buffer).
163dcbf3bdGeorge Wilson *
1645602294Dan Kimmel *   arc_buf_hdr_t
1655602294Dan Kimmel *   +-----------+
1665602294Dan Kimmel *   | fields    |
1675602294Dan Kimmel *   | common to |
1685602294Dan Kimmel *   | L1- and   |
1695602294Dan Kimmel *   | L2ARC     |
1705602294Dan Kimmel *   +-----------+
1715602294Dan Kimmel *   | l2arc_buf_hdr_t
1725602294Dan Kimmel *   |           |
1735602294Dan Kimmel *   +-----------+
1745602294Dan Kimmel *   | l1arc_buf_hdr_t
1755602294Dan Kimmel *   |           |              arc_buf_t
1765602294Dan Kimmel *   | b_buf     +------------>+-----------+      arc_buf_t
177770499eDan Kimmel *   | b_pabd    +-+           |b_next     +---->+-----------+
1785602294Dan Kimmel *   +-----------+ |           |-----------|     |b_next     +-->NULL
1795602294Dan Kimmel *                 |           |b_comp = T |     +-----------+
1805602294Dan Kimmel *                 |           |b_data     +-+   |b_comp = F |
1815602294Dan Kimmel *                 |           +-----------+ |   |b_data     +-+
1825602294Dan Kimmel *                 +->+------+               |   +-----------+ |
1835602294Dan Kimmel *        compressed  |      |               |                 |
1845602294Dan Kimmel *           data     |      |<--------------+                 | uncompressed
1855602294Dan Kimmel *                    +------+          compressed,            |     data
1865602294Dan Kimmel *                                        shared               +-->+------+
1875602294Dan Kimmel *                                         data                    |      |
1885602294Dan Kimmel *                                                                 |      |
1895602294Dan Kimmel *                                                                 +------+
190dcbf3bdGeorge Wilson *
191dcbf3bdGeorge Wilson * When a consumer reads a block, the ARC must first look to see if the
1925602294Dan Kimmel * arc_buf_hdr_t is cached. If the hdr is cached then the ARC allocates a new
1935602294Dan Kimmel * arc_buf_t and either copies uncompressed data into a new data buffer from an
194770499eDan Kimmel * existing uncompressed arc_buf_t, decompresses the hdr's b_pabd buffer into a
195770499eDan Kimmel * new data buffer, or shares the hdr's b_pabd buffer, depending on whether the
1965602294Dan Kimmel * hdr is compressed and the desired compression characteristics of the
1975602294Dan Kimmel * arc_buf_t consumer. If the arc_buf_t ends up sharing data with the
1985602294Dan Kimmel * arc_buf_hdr_t and both of them are uncompressed then the arc_buf_t must be
1995602294Dan Kimmel * the last buffer in the hdr's b_buf list, however a shared compressed buf can
2005602294Dan Kimmel * be anywhere in the hdr's list.
201dcbf3bdGeorge Wilson *
202dcbf3bdGeorge Wilson * The diagram below shows an example of an uncompressed ARC hdr that is
2035602294Dan Kimmel * sharing its data with an arc_buf_t (note that the shared uncompressed buf is
2045602294Dan Kimmel * the last element in the buf list):
205dcbf3bdGeorge Wilson *
206dcbf3bdGeorge Wilson *                arc_buf_hdr_t
207dcbf3bdGeorge Wilson *                +-----------+
208dcbf3bdGeorge Wilson *                |           |
209dcbf3bdGeorge Wilson *                |           |
210dcbf3bdGeorge Wilson *                |           |
211dcbf3bdGeorge Wilson *                +-----------+
212dcbf3bdGeorge Wilson * l2arc_buf_hdr_t|           |
213dcbf3bdGeorge Wilson *                |           |
214dcbf3bdGeorge Wilson *                +-----------+
215dcbf3bdGeorge Wilson * l1arc_buf_hdr_t|           |
216dcbf3bdGeorge Wilson *                |           |                 arc_buf_t    (shared)
217dcbf3bdGeorge Wilson *                |    b_buf  +------------>+---------+      arc_buf_t
218dcbf3bdGeorge Wilson *                |           |             |b_next   +---->+---------+
219770499eDan Kimmel *                |  b_pabd   +-+           |---------|     |b_next   +-->NULL
220dcbf3bdGeorge Wilson *                +-----------+ |           |         |     +---------+
221dcbf3bdGeorge Wilson *                              |           |b_data   +-+   |         |
222dcbf3bdGeorge Wilson *                              |           +---------+ |   |b_data   +-+
223dcbf3bdGeorge Wilson *                              +->+------+             |   +---------+ |
224dcbf3bdGeorge Wilson *                                 |      |             |               |
225dcbf3bdGeorge Wilson *                   uncompressed  |      |             |               |
226dcbf3bdGeorge Wilson *                        data     +------+             |               |
227dcbf3bdGeorge Wilson *                                    ^                 +->+------+     |
228dcbf3bdGeorge Wilson *                                    |       uncompressed |      |     |
229dcbf3bdGeorge Wilson *                                    |           data     |      |     |
230dcbf3bdGeorge Wilson *                                    |                    +------+     |
231dcbf3bdGeorge Wilson *                                    +---------------------------------+
232dcbf3bdGeorge Wilson *
233770499eDan Kimmel * Writing to the ARC requires that the ARC first discard the hdr's b_pabd
234dcbf3bdGeorge Wilson * since the physical block is about to be rewritten. The new data contents
2355602294Dan Kimmel * will be contained in the arc_buf_t. As the I/O pipeline performs the write,
2365602294Dan Kimmel * it may compress the data before writing it to disk. The ARC will be called
2375602294Dan Kimmel * with the transformed data and will bcopy the transformed on-disk block into
238770499eDan Kimmel * a newly allocated b_pabd. Writes are always done into buffers which have
2395602294Dan Kimmel * either been loaned (and hence are new and don't have other readers) or
2405602294Dan Kimmel * buffers which have been released (and hence have their own hdr, if there
2415602294Dan Kimmel * were originally other readers of the buf's original hdr). This ensures that
2425602294Dan Kimmel * the ARC only needs to update a single buf and its hdr after a write occurs.
243dcbf3bdGeorge Wilson *
244770499eDan Kimmel * When the L2ARC is in use, it will also take advantage of the b_pabd. The
245770499eDan Kimmel * L2ARC will always write the contents of b_pabd to the L2ARC. This means
2465602294Dan Kimmel * that when compressed ARC is enabled that the L2ARC blocks are identical
247dcbf3bdGeorge Wilson * to the on-disk block in the main data pool. This provides a significant
248dcbf3bdGeorge Wilson * advantage since the ARC can leverage the bp's checksum when reading from the
249dcbf3bdGeorge Wilson * L2ARC to determine if the contents are valid. However, if the compressed
2505602294Dan Kimmel * ARC is disabled, then the L2ARC's block must be transformed to look
251dcbf3bdGeorge Wilson * like the physical block in the main data pool before comparing the
252dcbf3bdGeorge Wilson * checksum and determining its validity.
253eb63303Tom Caputi *
254eb63303Tom Caputi * The L1ARC has a slightly different system for storing encrypted data.
255eb63303Tom Caputi * Raw (encrypted + possibly compressed) data has a few subtle differences from
256eb63303Tom Caputi * data that is just compressed. The biggest difference is that it is not
257eb63303Tom Caputi * possible to decrypt encrypted data (or visa versa) if the keys aren't loaded.
258eb63303Tom Caputi * The other difference is that encryption cannot be treated as a suggestion.
259eb63303Tom Caputi * If a caller would prefer compressed data, but they actually wind up with
260eb63303Tom Caputi * uncompressed data the worst thing that could happen is there might be a
261eb63303Tom Caputi * performance hit. If the caller requests encrypted data, however, we must be
262eb63303Tom Caputi * sure they actually get it or else secret information could be leaked. Raw
263eb63303Tom Caputi * data is stored in hdr->b_crypt_hdr.b_rabd. An encrypted header, therefore,
264eb63303Tom Caputi * may have both an encrypted version and a decrypted version of its data at
265eb63303Tom Caputi * once. When a caller needs a raw arc_buf_t, it is allocated and the data is
266eb63303Tom Caputi * copied out of this header. To avoid complications with b_pabd, raw buffers
267eb63303Tom Caputi * cannot be shared.
268dcbf3bdGeorge Wilson */
269dcbf3bdGeorge Wilson
270fa9e406ahrens#include <sys/spa.h>
271fa9e406ahrens#include <sys/zio.h>
272dcbf3bdGeorge Wilson#include <sys/spa_impl.h>
273aad0257Saso Kiselkov#include <sys/zio_compress.h>
274dcbf3bdGeorge Wilson#include <sys/zio_checksum.h>
275fa9e406ahrens#include <sys/zfs_context.h>
276fa9e406ahrens#include <sys/arc.h>
277fa9e406ahrens#include <sys/refcount.h>
278c5904d1eschrock#include <sys/vdev.h>
279573ca77George Wilson#include <sys/vdev_impl.h>
28069962b5Matthew Ahrens#include <sys/dsl_pool.h>
281770499eDan Kimmel#include <sys/zio_checksum.h>
282244781fPrakash Surya#include <sys/multilist.h>
283770499eDan Kimmel#include <sys/abd.h>
284eb63303Tom Caputi#include <sys/zil.h>
285eb63303Tom Caputi#include <sys/fm/fs/zfs.h>
286fa9e406ahrens#ifdef _KERNEL
287fa9e406ahrens#include <sys/vmsystm.h>
288fa9e406ahrens#include <vm/anon.h>
289fa9e406ahrens#include <sys/fs/swapnode.h>
290033f983ek#include <sys/dnlc.h>
292fa9e406ahrens#include <sys/callb.h>
29344cb6abbmc#include <sys/kstat.h>
294de753e3Brad Lewis#include <sys/zthr.h>
295b24ab67Jeff Bonwick#include <zfs_fletcher.h>
2963a2d8a1Paul Dagnelie#include <sys/aggsum.h>
2973a2d8a1Paul Dagnelie#include <sys/cityhash.h>
298af1d63aPaul Dagnelie#include <sys/param.h>
300cd1c8b8Matthew Ahrens#ifndef _KERNEL
301cd1c8b8Matthew Ahrens/* set with ZFS_DEBUG=watch, to enable watchpoints on frozen buffers */
302cd1c8b8Matthew Ahrensboolean_t arc_watch = B_FALSE;
303cd1c8b8Matthew Ahrensint arc_procfd;
304cd1c8b8Matthew Ahrens#endif
305cd1c8b8Matthew Ahrens
306de753e3Brad Lewis/*
307de753e3Brad Lewis * This thread's job is to keep enough free memory in the system, by
308de753e3Brad Lewis * calling arc_kmem_reap_now() plus arc_shrink(), which improves
309de753e3Brad Lewis * arc_available_memory().
310de753e3Brad Lewis */
311de753e3Brad Lewisstatic zthr_t		*arc_reap_zthr;
312de753e3Brad Lewis
313de753e3Brad Lewis/*
314de753e3Brad Lewis * This thread's job is to keep arc_size under arc_c, by calling
315de753e3Brad Lewis * arc_adjust(), which improves arc_is_overflowing().
316de753e3Brad Lewis */
317de753e3Brad Lewisstatic zthr_t		*arc_adjust_zthr;
318de753e3Brad Lewis
319de753e3Brad Lewisstatic kmutex_t		arc_adjust_lock;
320de753e3Brad Lewisstatic kcondvar_t	arc_adjust_waiters_cv;
321de753e3Brad Lewisstatic boolean_t	arc_adjust_needed = B_FALSE;
322244781fPrakash Surya
3232ec99e3Matthew Ahrensuint_t arc_reduce_dnlc_percent = 3;
32569962b5Matthew Ahrens/*
326244781fPrakash Surya * The number of headers to evict in arc_evict_state_impl() before
327244781fPrakash Surya * dropping the sublist lock and evicting from another sublist. A lower
328244781fPrakash Surya * value means we're more likely to evict the "correct" header (i.e. the
329244781fPrakash Surya * oldest header in the arc state), but comes with higher overhead
330244781fPrakash Surya * (i.e. more invocations of arc_evict_state_impl()).
331244781fPrakash Surya */
332244781fPrakash Suryaint zfs_arc_evict_batch_limit = 10;
333244781fPrakash Surya
334fa9e406ahrens/* number of seconds before growing cache again */
335de753e3Brad Lewisint arc_grow_retry = 60;
337de753e3Brad Lewis/*
338de753e3Brad Lewis * Minimum time between calls to arc_kmem_reap_soon().  Note that this will
339de753e3Brad Lewis * be converted to ticks, so with the default hz=100, a setting of 15 ms
340de753e3Brad Lewis * will actually wait 2 ticks, or 20ms.
341de753e3Brad Lewis */
342de753e3Brad Lewisint arc_kmem_cache_reap_retry_ms = 1000;
34336a64e6Tim Kordas
344770499eDan Kimmel/* shift of arc_c for calculating overflow limit in arc_get_data_impl */
345de753e3Brad Lewisint zfs_arc_overflow_shift = 8;
346244781fPrakash Surya
3475a98e54Brendan Gregg - Sun Microsystems/* shift of arc_c for calculating both min and max arc_p */
348de753e3Brad Lewisint arc_p_min_shift = 4;
3495a98e54Brendan Gregg - Sun Microsystems
3505a98e54Brendan Gregg - Sun Microsystems/* log2(fraction of arc to reclaim) */
351de753e3Brad Lewisint arc_shrink_shift = 7;
3522ec99e3Matthew Ahrens
3532ec99e3Matthew Ahrens/*
3542ec99e3Matthew Ahrens * log2(fraction of ARC which must be free to allow growing).
3552ec99e3Matthew Ahrens * I.e. If there is less than arc_c >> arc_no_grow_shift free memory,
3562ec99e3Matthew Ahrens * when reading a new block into the ARC, we will evict an equal-sized block
3572ec99e3Matthew Ahrens * from the ARC.
3582ec99e3Matthew Ahrens *
3592ec99e3Matthew Ahrens * This must be less than arc_shrink_shift, so that when we shrink the ARC,
3602ec99e3Matthew Ahrens * we will still not allow it to grow.
3612ec99e3Matthew Ahrens */
3622ec99e3Matthew Ahrensint			arc_no_grow_shift = 5;
3632ec99e3Matthew Ahrens
3645a98e54Brendan Gregg - Sun Microsystems
366b19a79eperrin * minimum lifespan of a prefetch block in clock ticks
367b19a79eperrin * (initialized in arc_init())
36813506d1maybee */
369a3874b8Toomas Soomestatic int		zfs_arc_min_prefetch_ms = 1;
370a3874b8Toomas Soomestatic int		zfs_arc_min_prescient_prefetch_ms = 6;
37269962b5Matthew Ahrens/*
37369962b5Matthew Ahrens * If this percent of memory is free, don't throttle.
37469962b5Matthew Ahrens */
37569962b5Matthew Ahrensint arc_lotsfree_percent = 10;
37669962b5Matthew Ahrens
377de753e3Brad Lewisstatic boolean_t arc_initialized;
3803a737e0brendan * The arc has filled available memory and has now warmed up.
3813a737e0brendan */
3823a737e0brendanstatic boolean_t arc_warm;
3850dd053dPrakash Surya * log2 fraction of the zio arena to keep free.
3860dd053dPrakash Surya */
3870dd053dPrakash Suryaint arc_zio_arena_free_shift = 2;
3880dd053dPrakash Surya
3890dd053dPrakash Surya/*
390a2eea2eahrens * These tunables are for performance analysis.
391a2eea2eahrens */
392a2eea2eahrensuint64_t zfs_arc_max;
393a2eea2eahrensuint64_t zfs_arc_min;
3941116048ekuint64_t zfs_arc_meta_limit = 0;
3953a5286aMatthew Ahrensuint64_t zfs_arc_meta_min = 0;
3965a98e54Brendan Gregg - Sun Microsystemsint zfs_arc_grow_retry = 0;
3975a98e54Brendan Gregg - Sun Microsystemsint zfs_arc_shrink_shift = 0;
3985a98e54Brendan Gregg - Sun Microsystemsint zfs_arc_p_min_shift = 0;
39963e911bMatthew Ahrensint zfs_arc_average_blocksize = 8 * 1024; /* 8KB */
401abe1fd0Don Brady/*
402abe1fd0Don Brady * ARC dirty data constraints for arc_tempreserve_space() throttle
403abe1fd0Don Brady */
404abe1fd0Don Bradyuint_t zfs_arc_dirty_limit_percent = 50;	/* total dirty data limit */
405abe1fd0Don Bradyuint_t zfs_arc_anon_limit_percent = 25;		/* anon block dirty limit */
406abe1fd0Don Bradyuint_t zfs_arc_pool_dirty_percent = 20;		/* each pool's anon allowance */
407abe1fd0Don Brady
408dcbf3bdGeorge Wilsonboolean_t zfs_compressed_arc_enabled = B_TRUE;
409dcbf3bdGeorge Wilson
411fa94a07brendan * Note that buffers can be in one of 6 states:
412fa9e406ahrens *	ARC_anon	- anonymous (discussed below)
413ea8dc4beschrock *	ARC_mru		- recently used, currently cached
414ea8dc4beschrock *	ARC_mru_ghost	- recentely used, no longer in cache
415ea8dc4beschrock *	ARC_mfu		- frequently used, currently cached
416ea8dc4beschrock *	ARC_mfu_ghost	- frequently used, no longer in cache
417fa94a07brendan *	ARC_l2c_only	- exists in L2ARC but not other states
4180e8c615maybee * When there are no active references to the buffer, they are
4190e8c615maybee * are linked onto a list in one of these arc states.  These are
4200e8c615maybee * the only buffers that can be evicted or deleted.  Within each
4210e8c615maybee * state there are multiple lists, one for meta-data and one for
4220e8c615maybee * non-meta-data.  Meta-data (indirect blocks, blocks of dnodes,
4230e8c615maybee * etc.) is tracked separately so that it can be managed more
424fa94a07brendan * explicitly: favored over data, limited explicitly.
425fa9e406ahrens *
426fa9e406ahrens * Anonymous buffers are buffers that are not associated with
427fa9e406ahrens * a DVA.  These are buffers that hold dirty block copies
428fa9e406ahrens * before they are written to stable storage.  By definition,
429ea8dc4beschrock * they are "ref'd" and are considered part of arc_mru
430fa9e406ahrens * that cannot be freed.  Generally, they will aquire a DVA
431ea8dc4beschrock * as they are written and migrate onto the arc_mru list.
432fa94a07brendan *
433fa94a07brendan * The ARC_l2c_only state is for buffers that are in the second
434fa94a07brendan * level ARC but no longer in any of the ARC_m* lists.  The second
435fa94a07brendan * level ARC itself may also contain buffers that are in any of
436fa94a07brendan * the ARC_m* states - meaning that a buffer can exist in two
437fa94a07brendan * places.  The reason for the ARC_l2c_only state is to keep the
438fa94a07brendan * buffer header in the hash table, so that reads that hit the
439fa94a07brendan * second level ARC benefit from these fast lookups.
440fa9e406ahrens */
442fa9e406ahrenstypedef struct arc_state {
443244781fPrakash Surya	/*
444244781fPrakash Surya	 * list of evictable buffers
445244781fPrakash Surya	 */
44694c2d0eMatthew Ahrens	multilist_t *arcs_list[ARC_BUFC_NUMTYPES];
447244781fPrakash Surya	/*
448244781fPrakash Surya	 * total amount of evictable data in this state
449244781fPrakash Surya	 */
450e914aceTim Schumacher	zfs_refcount_t arcs_esize[ARC_BUFC_NUMTYPES];
451244781fPrakash Surya	/*
452244781fPrakash Surya	 * total amount of data in this state; this includes: evictable,
453244781fPrakash Surya	 * non-evictable, ARC_BUFC_DATA, and ARC_BUFC_METADATA.
454244781fPrakash Surya	 */
455e914aceTim Schumacher	zfs_refcount_t arcs_size;
456fa9e406ahrens} arc_state_t;
458fa94a07brendan/* The 6 states: */
459fa9e406ahrensstatic arc_state_t ARC_anon;
460ea8dc4beschrockstatic arc_state_t ARC_mru;
461ea8dc4beschrockstatic arc_state_t ARC_mru_ghost;
462ea8dc4beschrockstatic arc_state_t ARC_mfu;
463ea8dc4beschrockstatic arc_state_t ARC_mfu_ghost;
464fa94a07brendanstatic arc_state_t ARC_l2c_only;
46644cb6abbmctypedef struct arc_stats {
46744cb6abbmc	kstat_named_t arcstat_hits;
46844cb6abbmc	kstat_named_t arcstat_misses;
46944cb6abbmc	kstat_named_t arcstat_demand_data_hits;
47044cb6abbmc	kstat_named_t arcstat_demand_data_misses;
47144cb6abbmc	kstat_named_t arcstat_demand_metadata_hits;
47244cb6abbmc	kstat_named_t arcstat_demand_metadata_misses;
47344cb6abbmc	kstat_named_t arcstat_prefetch_data_hits;
47444cb6abbmc	kstat_named_t arcstat_prefetch_data_misses;
47544cb6abbmc	kstat_named_t arcstat_prefetch_metadata_hits;
47644cb6abbmc	kstat_named_t arcstat_prefetch_metadata_misses;
47744cb6abbmc	kstat_named_t arcstat_mru_hits;
47844cb6abbmc	kstat_named_t arcstat_mru_ghost_hits;
47944cb6abbmc	kstat_named_t arcstat_mfu_hits;
48044cb6abbmc	kstat_named_t arcstat_mfu_ghost_hits;
48144cb6abbmc	kstat_named_t arcstat_deleted;
4823e30c24Will Andrews	/*
4833e30c24Will Andrews	 * Number of buffers that could not be evicted because the hash lock
4843e30c24Will Andrews	 * was held by another thread.  The lock may not necessarily be held
4853e30c24Will Andrews	 * by something using the same buffer, since hash locks are shared
4863e30c24Will Andrews	 * by multiple buffers.
4873e30c24Will Andrews	 */
48844cb6abbmc	kstat_named_t arcstat_mutex_miss;
4893e30c24Will Andrews	/*
4907b38fabAlexander Motin	 * Number of buffers skipped when updating the access state due to the
4917b38fabAlexander Motin	 * header having already been released after acquiring the hash lock.
4927b38fabAlexander Motin	 */
4937b38fabAlexander Motin	kstat_named_t arcstat_access_skip;
4947b38fabAlexander Motin	/*
4953e30c24Will Andrews	 * Number of buffers skipped because they have I/O in progress, are
4967b38fabAlexander Motin	 * indirect prefetch buffers that have not lived long enough, or are
4973e30c24Will Andrews	 * not from the spa we're trying to evict from.
4983e30c24Will Andrews	 */
49944cb6abbmc	kstat_named_t arcstat_evict_skip;
500244781fPrakash Surya	/*
501244781fPrakash Surya	 * Number of times arc_evict_state() was unable to evict enough
502eb63303Tom Caputi	 * buffers to reach its target amount.
503244781fPrakash Surya	 */
504244781fPrakash Surya	kstat_named_t arcstat_evict_not_enough;
5055ea40c0Brendan Gregg - Sun Microsystems	kstat_named_t arcstat_evict_l2_cached;
5065ea40c0Brendan Gregg - Sun Microsystems	kstat_named_t arcstat_evict_l2_eligible;
5075ea40c0Brendan Gregg - Sun Microsystems	kstat_named_t arcstat_evict_l2_ineligible;
508244781fPrakash Surya	kstat_named_t arcstat_evict_l2_skip;
50944cb6abbmc	kstat_named_t arcstat_hash_elements;
51044cb6abbmc	kstat_named_t arcstat_hash_elements_max;
51144cb6abbmc	kstat_named_t arcstat_hash_collisions;
51244cb6abbmc	kstat_named_t arcstat_hash_chains;
51344cb6abbmc	kstat_named_t arcstat_hash_chain_max;
51444cb6abbmc	kstat_named_t arcstat_p;
51544cb6abbmc	kstat_named_t arcstat_c;
51644cb6abbmc	kstat_named_t arcstat_c_min;
51744cb6abbmc	kstat_named_t arcstat_c_max;
5183a2d8a1Paul Dagnelie	/* Not updated directly; only synced in arc_kstat_update. */
51944cb6abbmc	kstat_named_t arcstat_size;
5204076b1bPrakash Surya	/*
521770499eDan Kimmel	 * Number of compressed bytes stored in the arc_buf_hdr_t's b_pabd.
522dcbf3bdGeorge Wilson	 * Note that the compressed bytes may match the uncompressed bytes
523dcbf3bdGeorge Wilson	 * if the block is either not compressed or compressed arc is disabled.
524dcbf3bdGeorge Wilson	 */
525dcbf3bdGeorge Wilson	kstat_named_t arcstat_compressed_size;
526dcbf3bdGeorge Wilson	/*
527770499eDan Kimmel	 * Uncompressed size of the data stored in b_pabd. If compressed
528dcbf3bdGeorge Wilson	 * arc is disabled then this value will be identical to the stat
529dcbf3bdGeorge Wilson	 * above.
530dcbf3bdGeorge Wilson	 */
531dcbf3bdGeorge Wilson	kstat_named_t arcstat_uncompressed_size;
532dcbf3bdGeorge Wilson	/*
533dcbf3bdGeorge Wilson	 * Number of bytes stored in all the arc_buf_t's. This is classified
534dcbf3bdGeorge Wilson	 * as "overhead" since this data is typically short-lived and will
535dcbf3bdGeorge Wilson	 * be evicted from the arc when it becomes unreferenced unless the
536dcbf3bdGeorge Wilson	 * zfs_keep_uncompressed_metadata or zfs_keep_uncompressed_level
537dcbf3bdGeorge Wilson	 * values have been set (see comment in dbuf.c for more information).
538dcbf3bdGeorge Wilson	 */
539dcbf3bdGeorge Wilson	kstat_named_t arcstat_overhead_size;
540dcbf3bdGeorge Wilson	/*
5414076b1bPrakash Surya	 * Number of bytes consumed by internal ARC structures necessary
5424076b1bPrakash Surya	 * for tracking purposes; these structures are not actually
5434076b1bPrakash Surya	 * backed by ARC buffers. This includes arc_buf_hdr_t structures
5444076b1bPrakash Surya	 * (allocated via arc_buf_hdr_t_full and arc_buf_hdr_t_l2only
5454076b1bPrakash Surya	 * caches), and arc_buf_t structures (allocated via arc_buf_t
5464076b1bPrakash Surya	 * cache).
5473a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
5484076b1bPrakash Surya	 */
549fa94a07brendan	kstat_named_t arcstat_hdr_size;
5504076b1bPrakash Surya	/*
5514076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers of type equal to
5524076b1bPrakash Surya	 * ARC_BUFC_DATA. This is generally consumed by buffers backing
5534076b1bPrakash Surya	 * on disk user data (e.g. plain file contents).
5543a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
5554076b1bPrakash Surya	 */
5565a98e54Brendan Gregg - Sun Microsystems	kstat_named_t arcstat_data_size;
5574076b1bPrakash Surya	/*
5584076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers of type equal to
5594076b1bPrakash Surya	 * ARC_BUFC_METADATA. This is generally consumed by buffers
5604076b1bPrakash Surya	 * backing on disk data that is used for internal ZFS
5614076b1bPrakash Surya	 * structures (e.g. ZAP, dnode, indirect blocks, etc).
5623a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
5634076b1bPrakash Surya	 */
5644076b1bPrakash Surya	kstat_named_t arcstat_metadata_size;
5654076b1bPrakash Surya	/*
5664076b1bPrakash Surya	 * Number of bytes consumed by various buffers and structures
5674076b1bPrakash Surya	 * not actually backed with ARC buffers. This includes bonus
5684076b1bPrakash Surya	 * buffers (allocated directly via zio_buf_* functions),
5694076b1bPrakash Surya	 * dmu_buf_impl_t structures (allocated via dmu_buf_impl_t
5704076b1bPrakash Surya	 * cache), and dnode_t structures (allocated via dnode_t cache).
5713a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
5724076b1bPrakash Surya	 */
5735a98e54Brendan Gregg - Sun Microsystems	kstat_named_t arcstat_other_size;
5744076b1bPrakash Surya	/*
5754076b1bPrakash Surya	 * Total number of bytes consumed by ARC buffers residing in the
5764076b1bPrakash Surya	 * arc_anon state. This includes *all* buffers in the arc_anon
5774076b1bPrakash Surya	 * state; e.g. data, metadata, evictable, and unevictable buffers
5784076b1bPrakash Surya	 * are all included in this value.
5793a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
5804076b1bPrakash Surya	 */
5814076b1bPrakash Surya	kstat_named_t arcstat_anon_size;
5824076b1bPrakash Surya	/*
5834076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers that meet the
5844076b1bPrakash Surya	 * following criteria: backing buffers of type ARC_BUFC_DATA,
5854076b1bPrakash Surya	 * residing in the arc_anon state, and are eligible for eviction
5864076b1bPrakash Surya	 * (e.g. have no outstanding holds on the buffer).
5873a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
5884076b1bPrakash Surya	 */
5894076b1bPrakash Surya	kstat_named_t arcstat_anon_evictable_data;
5904076b1bPrakash Surya	/*
5914076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers that meet the
5924076b1bPrakash Surya	 * following criteria: backing buffers of type ARC_BUFC_METADATA,
5934076b1bPrakash Surya	 * residing in the arc_anon state, and are eligible for eviction
5944076b1bPrakash Surya	 * (e.g. have no outstanding holds on the buffer).
5953a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
5964076b1bPrakash Surya	 */
5974076b1bPrakash Surya	kstat_named_t arcstat_anon_evictable_metadata;
5984076b1bPrakash Surya	/*
5994076b1bPrakash Surya	 * Total number of bytes consumed by ARC buffers residing in the
6004076b1bPrakash Surya	 * arc_mru state. This includes *all* buffers in the arc_mru
6014076b1bPrakash Surya	 * state; e.g. data, metadata, evictable, and unevictable buffers
6024076b1bPrakash Surya	 * are all included in this value.
6033a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6044076b1bPrakash Surya	 */
6054076b1bPrakash Surya	kstat_named_t arcstat_mru_size;
6064076b1bPrakash Surya	/*
6074076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers that meet the
6084076b1bPrakash Surya	 * following criteria: backing buffers of type ARC_BUFC_DATA,
6094076b1bPrakash Surya	 * residing in the arc_mru state, and are eligible for eviction
6104076b1bPrakash Surya	 * (e.g. have no outstanding holds on the buffer).
6113a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6124076b1bPrakash Surya	 */
6134076b1bPrakash Surya	kstat_named_t arcstat_mru_evictable_data;
6144076b1bPrakash Surya	/*
6154076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers that meet the
6164076b1bPrakash Surya	 * following criteria: backing buffers of type ARC_BUFC_METADATA,
6174076b1bPrakash Surya	 * residing in the arc_mru state, and are eligible for eviction
6184076b1bPrakash Surya	 * (e.g. have no outstanding holds on the buffer).
6193a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6204076b1bPrakash Surya	 */
6214076b1bPrakash Surya	kstat_named_t arcstat_mru_evictable_metadata;
6224076b1bPrakash Surya	/*
6234076b1bPrakash Surya	 * Total number of bytes that *would have been* consumed by ARC
6244076b1bPrakash Surya	 * buffers in the arc_mru_ghost state. The key thing to note
6254076b1bPrakash Surya	 * here, is the fact that this size doesn't actually indicate
6264076b1bPrakash Surya	 * RAM consumption. The ghost lists only consist of headers and
6274076b1bPrakash Surya	 * don't actually have ARC buffers linked off of these headers.
6284076b1bPrakash Surya	 * Thus, *if* the headers had associated ARC buffers, these
6294076b1bPrakash Surya	 * buffers *would have* consumed this number of bytes.
6303a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6314076b1bPrakash Surya	 */
6324076b1bPrakash Surya	kstat_named_t arcstat_mru_ghost_size;
6334076b1bPrakash Surya	/*
6344076b1bPrakash Surya	 * Number of bytes that *would have been* consumed by ARC
6354076b1bPrakash Surya	 * buffers that are eligible for eviction, of type
6364076b1bPrakash Surya	 * ARC_BUFC_DATA, and linked off the arc_mru_ghost state.
6373a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6384076b1bPrakash Surya	 */
6394076b1bPrakash Surya	kstat_named_t arcstat_mru_ghost_evictable_data;
6404076b1bPrakash Surya	/*
6414076b1bPrakash Surya	 * Number of bytes that *would have been* consumed by ARC
6424076b1bPrakash Surya	 * buffers that are eligible for eviction, of type
6434076b1bPrakash Surya	 * ARC_BUFC_METADATA, and linked off the arc_mru_ghost state.
6443a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6454076b1bPrakash Surya	 */
6464076b1bPrakash Surya	kstat_named_t arcstat_mru_ghost_evictable_metadata;
6474076b1bPrakash Surya	/*
6484076b1bPrakash Surya	 * Total number of bytes consumed by ARC buffers residing in the
6494076b1bPrakash Surya	 * arc_mfu state. This includes *all* buffers in the arc_mfu
6504076b1bPrakash Surya	 * state; e.g. data, metadata, evictable, and unevictable buffers
6514076b1bPrakash Surya	 * are all included in this value.
6523a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6534076b1bPrakash Surya	 */
6544076b1bPrakash Surya	kstat_named_t arcstat_mfu_size;
6554076b1bPrakash Surya	/*
6564076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers that are eligible for
6574076b1bPrakash Surya	 * eviction, of type ARC_BUFC_DATA, and reside in the arc_mfu
6584076b1bPrakash Surya	 * state.
6593a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6604076b1bPrakash Surya	 */
6614076b1bPrakash Surya	kstat_named_t arcstat_mfu_evictable_data;
6624076b1bPrakash Surya	/*
6634076b1bPrakash Surya	 * Number of bytes consumed by ARC buffers that are eligible for
6644076b1bPrakash Surya	 * eviction, of type ARC_BUFC_METADATA, and reside in the
6654076b1bPrakash Surya	 * arc_mfu state.
6663a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6674076b1bPrakash Surya	 */
6684076b1bPrakash Surya	kstat_named_t arcstat_mfu_evictable_metadata;
6694076b1bPrakash Surya	/*
6704076b1bPrakash Surya	 * Total number of bytes that *would have been* consumed by ARC
6714076b1bPrakash Surya	 * buffers in the arc_mfu_ghost state. See the comment above
6724076b1bPrakash Surya	 * arcstat_mru_ghost_size for more details.
6733a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6744076b1bPrakash Surya	 */
6754076b1bPrakash Surya	kstat_named_t arcstat_mfu_ghost_size;
6764076b1bPrakash Surya	/*
6774076b1bPrakash Surya	 * Number of bytes that *would have been* consumed by ARC
6784076b1bPrakash Surya	 * buffers that are eligible for eviction, of type
6794076b1bPrakash Surya	 * ARC_BUFC_DATA, and linked off the arc_mfu_ghost state.
6803a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6814076b1bPrakash Surya	 */
6824076b1bPrakash Surya	kstat_named_t arcstat_mfu_ghost_evictable_data;
6834076b1bPrakash Surya	/*
6844076b1bPrakash Surya	 * Number of bytes that *would have been* consumed by ARC
6854076b1bPrakash Surya	 * buffers that are eligible for eviction, of type
6864076b1bPrakash Surya	 * ARC_BUFC_METADATA, and linked off the arc_mru_ghost state.
6873a2d8a1Paul Dagnelie	 * Not updated directly; only synced in arc_kstat_update.
6884076b1bPrakash Surya	 */
6894076b1bPrakash Surya	kstat_named_t arcstat_mfu_ghost_evictable_metadata;
690fa94a07brendan	kstat_named_t arcstat_l2_hits;
691fa94a07brendan	kstat_named_t arcstat_l2_misses;
692fa94a07brendan	kstat_named_t arcstat_l2_feeds;
693fa94a07brendan	kstat_named_t arcstat_l2_rw_clash;
6945a98e54Brendan Gregg - Sun Microsystems	kstat_named_t arcstat_l2_read_bytes;
6955a98e54Brendan Gregg - Sun Microsystems	kstat_named_t arcstat_l2_write_bytes;
696fa94a07brendan	kstat_named_t arcstat_l2_writes_sent;
697fa94a07brendan	kstat_named_t arcstat_l2_writes_done;
698fa94a07brendan	kstat_named_t arcstat_l2_writes_error;
699244781fPrakash Surya	kstat_named_t arcstat_l2_writes_lock_retry;
700fa94a07brendan	kstat_named_t arcstat_l2_evict_lock_retry;
701fa94a07brendan	kstat_named_t arcstat_l2_evict_reading;
70289c86e3Chris Williamson	kstat_named_t arcstat_l2_evict_l1cached;
703fa94a07brendan	kstat_named_t arcstat_l2_free_on_write;
704fa94a07brendan	kstat_named_t arcstat_l2_abort_lowmem;
705fa94a07brendan	kstat_named_t arcstat_l2_cksum_bad;
706fa94a07brendan	kstat_named_t arcstat_l2_io_error;
70716a7e5aAndriy Gapon	kstat_named_t arcstat_l2_lsize;
70816a7e5aAndriy Gapon	kstat_named_t arcstat_l2_psize;
7093a2d8a1Paul Dagnelie	/* Not updated directly; only synced in arc_kstat_update. */
710fa94a07brendan	kstat_named_t arcstat_l2_hdr_size;
7111ab7f2dmaybee	kstat_named_t arcstat_memory_throttle_count;
7123a2d8a1Paul Dagnelie	/* Not updated directly; only synced in arc_kstat_update. */
71320128a0George Wilson	kstat_named_t arcstat_meta_used;
71420128a0George Wilson	kstat_named_t arcstat_meta_limit;
71520128a0George Wilson	kstat_named_t arcstat_meta_max;
7163a5286aMatthew Ahrens	kstat_named_t arcstat_meta_min;
717a3874b8Toomas Soome	kstat_named_t arcstat_async_upgrade_sync;
718cf6106cMatthew Ahrens	kstat_named_t arcstat_demand_hit_predictive_prefetch;
719a3874b8Toomas Soome	kstat_named_t arcstat_demand_hit_prescient_prefetch;
72044cb6abbmc} arc_stats_t;
72244cb6abbmcstatic arc_stats_t arc_stats = {
72344cb6abbmc	{ "hits",			KSTAT_DATA_UINT64 },
72444cb6abbmc	{ "misses",			KSTAT_DATA_UINT64 },
72544cb6abbmc	{ "demand_data_hits",		KSTAT_DATA_UINT64 },
72644cb6abbmc	{ "demand_data_misses",		KSTAT_DATA_UINT64 },
72744cb6abbmc	{ "demand_metadata_hits",	KSTAT_DATA_UINT64 },
72844cb6abbmc	{ "demand_metadata_misses",	KSTAT_DATA_UINT64 },
72944cb6abbmc	{ "prefetch_data_hits",		KSTAT_DATA_UINT64 },
73044cb6abbmc	{ "prefetch_data_misses",	KSTAT_DATA_UINT64 },
73144cb6abbmc	{ "prefetch_metadata_hits",	KSTAT_DATA_UINT64 },
73244cb6abbmc	{ "prefetch_metadata_misses",	KSTAT_DATA_UINT64 },
73344cb6abbmc	{ "mru_hits",			KSTAT_DATA_UINT64 },
73444cb6abbmc	{ "mru_ghost_hits",		KSTAT_DATA_UINT64 },
73544cb6abbmc	{ "mfu_hits",			KSTAT_DATA_UINT64 },
73644cb6abbmc	{ "mfu_ghost_hits",		KSTAT_DATA_UINT64 },
73744cb6abbmc	{ "deleted",			KSTAT_DATA_UINT64 },
73844cb6abbmc	{ "mutex_miss",			KSTAT_DATA_UINT64 },
7397b38fabAlexander Motin	{ "access_skip",		KSTAT_DATA_UINT64 },
74044cb6abbmc	{ "evict_skip",			KSTAT_DATA_UINT64 },
741244781fPrakash Surya	{ "evict_not_enough",		KSTAT_DATA_UINT64 },
7425ea40c0Brendan Gregg - Sun Microsystems	{ "evict_l2_cached",		KSTAT_DATA_UINT64 },
7435ea40c0Brendan Gregg - Sun Microsystems	{ "evict_l2_eligible",		KSTAT_DATA_UINT64 },
7445ea40c0Brendan Gregg - Sun Microsystems	{ "evict_l2_ineligible",	KSTAT_DATA_UINT64 },
745244781fPrakash Surya	{ "evict_l2_skip",		KSTAT_DATA_UINT64 },
74644cb6abbmc	{ "hash_elements",		KSTAT_DATA_UINT64 },
74744cb6abbmc	{ "hash_elements_max",		KSTAT_DATA_UINT64 },
74844cb6abbmc	{ "hash_collisions",		KSTAT_DATA_UINT64 },
74944cb6abbmc	{ "hash_chains",		KSTAT_DATA_UINT64 },
75044cb6abbmc	{ "hash_chain_max",		KSTAT_DATA_UINT64 },
75144cb6abbmc	{ "p",				KSTAT_DATA_UINT64 },
75244cb6abbmc	{ "c",				KSTAT_DATA_UINT64 },
75344cb6abbmc	{ "c_min",			KSTAT_DATA_UINT64 },
75444cb6abbmc	{ "c_max",			KSTAT_DATA_UINT64 },
755fa94a07brendan	{ "size",			KSTAT_DATA_UINT64 },
756dcbf3bdGeorge Wilson	{ "compressed_size",		KSTAT_DATA_UINT64 },
757dcbf3bdGeorge Wilson	{ "uncompressed_size",		KSTAT_DATA_UINT64 },
758dcbf3bdGeorge Wilson	{ "overhead_size",		KSTAT_DATA_UINT64 },
759fa94a07brendan	{ "hdr_size",			KSTAT_DATA_UINT64 },
7605a98e54Brendan Gregg - Sun Microsystems	{ "data_size",			KSTAT_DATA_UINT64 },
7614076b1bPrakash Surya	{ "metadata_size",		KSTAT_DATA_UINT64 },
7625a98e54Brendan Gregg - Sun Microsystems	{ "other_size",			KSTAT_DATA_UINT64 },
7634076b1bPrakash Surya	{ "anon_size",			KSTAT_DATA_UINT64 },
7644076b1bPrakash Surya	{ "anon_evictable_data",	KSTAT_DATA_UINT64 },
7654076b1bPrakash Surya	{ "anon_evictable_metadata",	KSTAT_DATA_UINT64 },
7664076b1bPrakash Surya	{ "mru_size",			KSTAT_DATA_UINT64 },
7674076b1bPrakash Surya	{ "mru_evictable_data",		KSTAT_DATA_UINT64 },
7684076b1bPrakash Surya	{ "mru_evictable_metadata",	KSTAT_DATA_UINT64 },
7694076b1bPrakash Surya	{ "mru_ghost_size",		KSTAT_DATA_UINT64 },
7704076b1bPrakash Surya	{ "mru_ghost_evictable_data",	KSTAT_DATA_UINT64 },
7714076b1bPrakash Surya	{ "mru_ghost_evictable_metadata", KSTAT_DATA_UINT64 },
7724076b1bPrakash Surya	{ "mfu_size",			KSTAT_DATA_UINT64 },