direct-io.hg

changeset 10588:eb6b0c7d9992

[IA64] add comments on smp

add notes on smp to mm.c

Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
author awilliam@xenbuild.aw
date Mon Jun 19 13:15:31 2006 -0600 (2006-06-19)
parents 5389c7b06ccf
children b16063ae0703
files xen/arch/ia64/xen/mm.c
line diff
     1.1 --- a/xen/arch/ia64/xen/mm.c	Mon Jun 19 13:13:01 2006 -0600
     1.2 +++ b/xen/arch/ia64/xen/mm.c	Mon Jun 19 13:15:31 2006 -0600
     1.3 @@ -9,6 +9,159 @@
     1.4   *                    dom0 vp model support
     1.5   */
     1.6  
     1.7 +/*
     1.8 + * NOTES on SMP
     1.9 + * 
    1.10 + * * shared structures
    1.11 + * There are some structures which are accessed by CPUs concurrently.
    1.12 + * Here is the list of shared structures and operations on them which
    1.13 + * read/write the structures.
    1.14 + * 
    1.15 + * - struct page_info
    1.16 + *   This is a xen global resource. This structure is accessed by
    1.17 + *   any CPUs.
    1.18 + * 
    1.19 + *   operations on this structure:
    1.20 + *   - get_page() and its variant
    1.21 + *   - put_page() and its variant
    1.22 + * 
    1.23 + * - vTLB
    1.24 + *   vcpu->arch.{d, i}tlb: Software tlb cache. These are per VCPU data.
    1.25 + *   DEFINE_PER_CPU (unsigned long, vhpt_paddr): VHPT table per physical CPU.
    1.26 + * 
    1.27 + *   domain_flush_vtlb_range() and domain_flush_vtlb_all()
    1.28 + *   write vcpu->arch.{d, i}tlb and VHPT table of vcpu which isn't current.
    1.29 + *   So there are potential races to read/write VHPT and vcpu->arch.{d, i}tlb.
    1.30 + *   Please note that reading VHPT is done by hardware page table walker.
    1.31 + * 
    1.32 + *   operations on this structure:
    1.33 + *   - global tlb purge
    1.34 + *     vcpu_ptc_g(), vcpu_ptc_ga() and domain_page_flush()
    1.35 + *     I.e. callers of domain_flush_vtlb_range() and domain_flush_vtlb_all()
    1.36 + *     These functions invalidate VHPT entry and vcpu->arch.{i, d}tlb
    1.37 + * 
    1.38 + *   - tlb insert and fc
    1.39 + *     vcpu_itc_i()
    1.40 + *     vcpu_itc_d()
    1.41 + *     ia64_do_page_fault()
    1.42 + *     vcpu_fc()
    1.43 + *     These functions set VHPT entry and vcpu->arch.{i, d}tlb.
    1.44 + *     Actually vcpu_itc_no_srlz() does.
    1.45 + * 
    1.46 + * - the P2M table
    1.47 + *   domain->mm and pgd, pud, pmd, pte table page.
    1.48 + *   This structure is used to convert domain pseudo physical address
    1.49 + *   to machine address. This is per domain resource.
    1.50 + * 
    1.51 + *   operations on this structure:
    1.52 + *   - populate the P2M table tree
    1.53 + *     lookup_alloc_domain_pte() and its variants.
    1.54 + *   - set p2m entry
    1.55 + *     assign_new_domain_page() and its variants.
    1.56 + *     assign_domain_page() and its variants.
    1.57 + *   - xchg p2m entry
    1.58 + *     assign_domain_page_replace()
    1.59 + *   - cmpxchg p2m entry
    1.60 + *     assign_domain_page_cmpxchg_rel()
    1.61 + *     destroy_grant_host_mapping()
    1.62 + *     steal_page_for_grant_transfer()
    1.63 + *     zap_domain_page_one()
    1.64 + *   - read p2m entry
    1.65 + *     lookup_alloc_domain_pte() and its variants.
    1.66 + *     
    1.67 + * - the M2P table
    1.68 + *   mpt_table (or machine_to_phys_mapping)
    1.69 + *   This is a table which converts from machine address to pseudo physical
    1.70 + *   address. This is a global structure.
    1.71 + * 
    1.72 + *   operations on this structure:
    1.73 + *   - set m2p entry
    1.74 + *     set_gpfn_from_mfn()
    1.75 + *   - zap m2p entry
    1.76 + *     set_gpfn_from_mfn(INVALID_P2M_ENTRY)
    1.77 + *   - get m2p entry
    1.78 + *     get_gpfn_from_mfn()
    1.79 + * 
    1.80 + * 
    1.81 + * * avoiding races
    1.82 + * The resources which are shared by CPUs must be accessed carefully
    1.83 + * to avoid race.
    1.84 + * IA64 has weak memory ordering so that attention must be paid
    1.85 + * to access shared structures. [SDM vol2 PartII chap. 2]
    1.86 + * 
    1.87 + * - struct page_info memory ordering
    1.88 + *   get_page() has acquire semantics.
    1.89 + *   put_page() has release semantics.
    1.90 + * 
    1.91 + * - populating the p2m table
    1.92 + *   pgd, pud, pmd are append only.
    1.93 + * 
    1.94 + * - races when updating the P2M tables and the M2P table
    1.95 + *   The P2M entry are shared by more than one vcpu.
    1.96 + *   So they are accessed atomic operations.
    1.97 + *   I.e. xchg or cmpxchg must be used to update the p2m entry.
    1.98 + *   NOTE: When creating/destructing a domain, we don't need to take care of
    1.99 + *         this race.
   1.100 + * 
   1.101 + *   The M2P table is inverse of the P2M table.
   1.102 + *   I.e. P2M(M2P(p)) = p and M2P(P2M(m)) = m
   1.103 + *   The M2P table and P2M table must be updated consistently.
   1.104 + *   Here is the update sequence
   1.105 + * 
   1.106 + *   xchg or cmpxchg case
   1.107 + *   - set_gpfn_from_mfn(new_mfn, gpfn)
   1.108 + *   - memory barrier
   1.109 + *   - atomic update of the p2m entry (xchg or cmpxchg the p2m entry)
   1.110 + *     get old_mfn entry as a result.
   1.111 + *   - memory barrier
   1.112 + *   - set_gpfn_from_mfn(old_mfn, INVALID_P2M_ENTRY)
   1.113 + * 
   1.114 + *   Here memory barrier can be achieved by release semantics.
   1.115 + * 
   1.116 + * - races between global tlb purge and tlb insert
   1.117 + *   This is a race between reading/writing vcpu->arch.{d, i}tlb or VHPT entry.
   1.118 + *   When a vcpu is about to insert tlb, another vcpu may purge tlb
   1.119 + *   cache globally. Inserting tlb (vcpu_itc_no_srlz()) or global tlb purge
   1.120 + *   (domain_flush_vtlb_range() and domain_flush_vtlb_all()) can't update
   1.121 + *   cpu->arch.{d, i}tlb, VHPT and mTLB. So there is a race here.
   1.122 + * 
   1.123 + *   Here check vcpu->arch.{d, i}tlb.p bit
   1.124 + *   After inserting tlb entry, check the p bit and retry to insert.
   1.125 + *   This means that when global tlb purge and tlb insert are issued
   1.126 + *   simultaneously, always global tlb purge happens after tlb insert.
   1.127 + * 
   1.128 + * - races between p2m entry update and tlb insert
   1.129 + *   This is a race between reading/writing the p2m entry.
   1.130 + *   reader: vcpu_itc_i(), vcpu_itc_d(), ia64_do_page_fault(), vcpu_fc()
   1.131 + *   writer: assign_domain_page_cmpxchg_rel(), destroy_grant_host_mapping(), 
   1.132 + *           steal_page_for_grant_transfer(), zap_domain_page_one()
   1.133 + * 
   1.134 + *   For example, vcpu_itc_i() is about to insert tlb by calling
   1.135 + *   vcpu_itc_no_srlz() after reading the p2m entry.
   1.136 + *   At the same time, the p2m entry is replaced by xchg or cmpxchg and
   1.137 + *   tlb cache of the page is flushed.
   1.138 + *   There is a possibility that the p2m entry doesn't already point to the
   1.139 + *   old page, but tlb cache still points to the old page.
   1.140 + *   This can be detected similar to sequence lock using the p2m entry itself.
   1.141 + *   reader remember the read value of the p2m entry, and insert tlb.
   1.142 + *   Then read the p2m entry again. If the new p2m entry value is different
   1.143 + *   from the used p2m entry value, the retry.
   1.144 + * 
   1.145 + * - races between referencing page and p2m entry update
   1.146 + *   This is a race between reading/writing the p2m entry.
   1.147 + *   reader: vcpu_get_domain_bundle(), vmx_get_domain_bundle(),
   1.148 + *           efi_emulate_get_time()
   1.149 + *   writer: assign_domain_page_cmpxchg_rel(), destroy_grant_host_mapping(), 
   1.150 + *           steal_page_for_grant_transfer(), zap_domain_page_one()
   1.151 + * 
   1.152 + *   A page which assigned to a domain can be de-assigned by another vcpu.
   1.153 + *   So before read/write to a domain page, the page's reference count 
   1.154 + *   must be incremented.
   1.155 + *   vcpu_get_domain_bundle(), vmx_get_domain_bundle() and
   1.156 + *   efi_emulate_get_time()
   1.157 + * 
   1.158 + */
   1.159 +
   1.160  #include <xen/config.h>
   1.161  #include <xen/sched.h>
   1.162  #include <xen/domain.h>